2023-11-28T14:16:23.686 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T14:16:23.687 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} --owner scheduled_matan@teuthology --name matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175 --verbose -- /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175/orig.config.yaml 2023-11-28T14:16:23.739 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T14:16:23.817 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175 branch: wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} email: null first_in_suite: false job_id: '7470175' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: b9b6c42eed672399c3d01aa72f6cda1a665a3c35 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9581 sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: b9b6c42eed672399c3d01aa72f6cda1a665a3c35 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBTRdcTUMH9vNDqA18WeFcFNIb7sxkBy1gegmQd57B94t5Nr/JXr4SFs5KEqqXIDvjAFecctn0ydU67mzIyR+r4= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_14:04:23 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T14:16:23.817 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa; will attempt to use it 2023-11-28T14:16:23.818 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa/tasks 2023-11-28T14:16:23.818 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T14:16:23.820 INFO:teuthology.task.internal:Checking packages... 2023-11-28T14:16:23.845 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '5537419ca37a48f0ea28d7872ae1f14ccf1a953b' 2023-11-28T14:16:23.845 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T14:16:23.845 INFO:teuthology.packaging:ref: None 2023-11-28T14:16:23.845 INFO:teuthology.packaging:tag: None 2023-11-28T14:16:23.845 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing 2023-11-28T14:16:23.845 INFO:teuthology.packaging:sha1: 5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:16:23.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing 2023-11-28T14:16:24.078 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7503.g5537419c 2023-11-28T14:16:24.079 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T14:16:24.086 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T14:16:24.086 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T14:16:24.163 INFO:teuthology.task.internal:Saving configuration 2023-11-28T14:16:24.185 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T14:16:24.193 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T14:16:24.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 14:11:39.555596', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBTRdcTUMH9vNDqA18WeFcFNIb7sxkBy1gegmQd57B94t5Nr/JXr4SFs5KEqqXIDvjAFecctn0ydU67mzIyR+r4='} 2023-11-28T14:16:24.219 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T14:16:24.227 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-28T14:16:24.227 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T14:16:24.270 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f05822319d0>, signals=[15]) 2023-11-28T14:16:24.270 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T14:16:24.277 INFO:teuthology.task.internal:Opening connections... 2023-11-28T14:16:24.281 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:16:24.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:16:24.383 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T14:16:24.392 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2023-11-28T14:16:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2023-11-28T14:16:24.452 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2023-11-28T14:16:24.511 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2023-11-28T14:16:24.511 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2023-11-28T14:16:24.511 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2023-11-28T14:16:24.511 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2023-11-28T14:16:24.512 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2023-11-28T14:16:24.512 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2023-11-28T14:16:24.512 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T14:16:24.512 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2023-11-28T14:16:24.512 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2023-11-28T14:16:24.513 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T14:16:24.513 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2023-11-28T14:16:24.513 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T14:16:24.514 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T14:16:24.515 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T14:16:24.517 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2023-11-28T14:16:24.542 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T14:16:24.553 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T14:16:24.601 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T14:16:24.601 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2023-11-28T14:16:24.618 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T14:16:24.626 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T14:16:24.626 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2023-11-28T14:16:24.684 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T14:16:24.716 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T14:16:24.731 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-28T14:16:24.731 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-28T14:16:24.732 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T14:16:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:16:24.750 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2023-11-28T14:16:24.807 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-361.el9.x86_64 2023-11-28T14:16:24.807 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-361.el9.x86_64 2023-11-28T14:16:24.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2023-11-28T14:16:26.864 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-11-28T14:16:28.248 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-11-28T14:16:30.025 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-11-28T14:16:33.464 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 18 MB 00:02 2023-11-28T14:16:38.190 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 5.8 MB 00:01 2023-11-28T14:16:39.625 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 15 kB 00:00 2023-11-28T14:16:41.102 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-11-28T14:16:46.026 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-28T14:16:48.441 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-28T14:16:48.441 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-28T14:16:48.526 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:16:48.531 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:16:48.531 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-28T14:16:48.531 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:16:48.531 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:16:48.531 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-386.el9 baseos 5.5 M 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-386.el9 baseos 20 M 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-386.el9 baseos 38 M 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-386.el9 baseos 32 M 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:16:48.532 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:16:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2023-11-28T14:16:48.533 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:16:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 95 M 2023-11-28T14:16:48.533 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 121 M 2023-11-28T14:16:48.534 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:16:49.969 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-386.el9.x86_64.rpm 4.0 MB/s | 5.5 MB 00:01 2023-11-28T14:16:53.370 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-modules-5.14.0-386.el9.x86_64.rpm 7.9 MB/s | 38 MB 00:04 2023-11-28T14:16:53.678 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-modules-core-5.14.0-386.el9.x86_6 8.7 MB/s | 32 MB 00:03 2023-11-28T14:16:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-core-5.14.0-386.el9.x86_64.rpm 3.2 MB/s | 20 MB 00:06 2023-11-28T14:16:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:16:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:Total 15 MB/s | 95 MB 00:06 2023-11-28T14:16:55.841 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:16:56.259 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:16:56.259 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:16:57.667 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:16:57.667 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:17:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:17:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-386.el9.x86_64 1/4 2023-11-28T14:17:01.305 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:17:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:17:04.370 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:17:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:17:09.454 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:17:14.433 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:17:44.743 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:17:44.788 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:17:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:17:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-386.el9.x86_64 1/4 2023-11-28T14:17:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-28T14:17:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-28T14:17:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-28T14:17:45.667 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-386.el9.x86_64 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-386.el9.x86_64 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-386.el9.x86_64 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-386.el9.x86_64 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:17:45.668 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:17:46.164 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T14:17:46.327 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-28T14:17:46.327 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-28T14:17:46.327 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-386.el9.x86_64 2023-11-28T14:17:46.328 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-28T14:17:46.328 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-28T14:17:46.328 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-28T14:17:46.328 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2023-11-28T14:17:46.328 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T14:17:46.452 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-28T14:17:46.452 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-28T14:17:46.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2023-11-28T14:17:46.491 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2023-11-28T14:17:46.491 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2023-11-28T14:17:46.491 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2023-11-28T14:17:46.491 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 68.el9 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8264254 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2023-11-28T14:17:46.492 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:17:46.493 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T14:17:46.494 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T14:17:46.495 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-386.el9.x86_64 2023-11-28T14:17:46.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T14:17:47.143 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2023-11-28T14:17:48.577 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T14:17:48.605 INFO:teuthology.orchestra.run.smithi038.stderr:done 2023-11-28T14:17:48.607 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2023-11-28T14:17:48.662 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.oXZJyDlvKt 2023-11-28T14:17:48.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oXZJyDlvKt 2023-11-28T14:17:48.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.oXZJyDlvKt 2023-11-28T14:17:48.883 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.oXZJyDlvKt is 6KB 2023-11-28T14:17:48.932 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.oXZJyDlvKt 2023-11-28T14:17:48.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T14:17:49.016 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T14:17:49.016 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T14:17:49.016 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T14:17:49.016 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64.conf 2023-11-28T14:17:49.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64 2023-11-28T14:17:49.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2023-11-28T14:17:49.277 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-11-28T14:18:19.212 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-28T14:18:19.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-28T14:18:19.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:18:37.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-11-28T14:18:46.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-28T14:18:46.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:19:46.606 DEBUG:teuthology.orchestra.remote:timed out 2023-11-28T14:19:58.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-28T14:19:58.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:19:58.923 DEBUG:teuthology.orchestra.run.smithi038:> true 2023-11-28T14:19:59.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-28T14:19:59.294 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-386.el9.x86_64"... 2023-11-28T14:19:59.294 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2023-11-28T14:19:59.310 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-386.el9.x86_64 2023-11-28T14:19:59.311 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-386.el9.x86_64 vs 5.14.0-386.el9.x86_64 2023-11-28T14:19:59.311 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T14:19:59.311 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T14:20:00.313 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-28T14:20:00.313 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T14:20:00.396 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2023-11-28T14:20:00.440 DEBUG:teuthology.parallel:result is None 2023-11-28T14:20:00.440 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T14:20:00.449 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T14:20:00.449 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T14:20:00.466 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T14:20:00.473 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T14:20:00.482 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T14:20:00.482 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T14:20:00.541 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T14:20:00.550 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T14:20:00.550 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T14:20:00.621 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T14:20:00.638 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T14:20:00.640 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T14:20:00.647 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T14:20:00.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T14:20:00.679 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T14:20:00.688 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T14:20:00.688 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T14:20:00.736 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T14:20:00.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T14:20:00.936 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T14:20:00.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T14:20:01.024 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:20:01.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T14:20:01.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2023-11-28T14:20:01.187 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T14:20:01.217 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T14:20:01.225 INFO:teuthology.task.internal:Starting timer... 2023-11-28T14:20:01.225 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T14:20:01.236 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T14:20:01.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2023-11-28T14:20:01.312 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2023-11-28T14:20:01.313 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T14:20:01.313 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2023-11-28T14:20:01.338 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2023-11-28T14:20:01.338 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive'} 2023-11-28T14:20:01.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T14:20:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:20:01.410 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T14:20:01.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2023-11-28T14:20:01.489 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T14:20:01.500 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-28T14:20:01.504 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-28T14:20:01.630 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T14:20:01.646 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T14:20:01.647 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T14:25:53.502 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com')] 2023-11-28T14:25:53.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-28T14:25:53.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T14:25:53.605 DEBUG:teuthology.orchestra.run.smithi038:> true 2023-11-28T14:25:53.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-11-28T14:25:53.674 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T14:25:53.685 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T14:25:53.685 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T14:25:53.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T14:25:53.750 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T14:25:53.770 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T14:25:53.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2023-11-28T14:25:53.823 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2023-11-28T14:25:53.843 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T14:25:53.863 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T14:25:53.911 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2023-11-28T14:25:53.914 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T14:25:53.914 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2023-11-28T14:25:53.914 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:25:53.914 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:25:53.914 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:25:53.914 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:25:53.915 INFO:teuthology.run_tasks:Running task install... 2023-11-28T14:25:53.925 DEBUG:teuthology.task.install:project ceph 2023-11-28T14:25:53.925 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'}} 2023-11-28T14:25:53.925 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'} 2023-11-28T14:25:53.925 INFO:teuthology.task.install:Using flavor: crimson 2023-11-28T14:25:53.933 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T14:25:53.934 INFO:teuthology.task.install:extra packages: [] 2023-11-28T14:25:53.934 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'tag': None, 'wait_for_package': False} 2023-11-28T14:25:53.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:25:54.150 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/ 2023-11-28T14:25:54.151 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7503.g5537419c 2023-11-28T14:25:54.424 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T14:25:54.424 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:25:54.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T14:25:54.460 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-28T14:25:54.461 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5537419ca37a48f0ea28d7872ae1f14ccf1a953b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T14:25:54.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T14:25:54.626 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T14:25:54.658 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2023-11-28T14:25:54.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2023-11-28T14:25:54.914 INFO:teuthology.orchestra.run.smithi038.stdout:73 files removed 2023-11-28T14:25:54.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2023-11-28T14:25:56.219 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2023-11-28T14:25:56.979 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 13 kB/s | 9.5 kB 00:00 2023-11-28T14:25:57.712 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 2.8 kB/s | 2.0 kB 00:00 2023-11-28T14:25:58.148 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-11-28T14:25:59.531 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-11-28T14:26:01.273 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 7.9 MB 00:01 2023-11-28T14:26:05.648 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 18 MB 00:03 2023-11-28T14:26:10.757 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.8 MB 00:01 2023-11-28T14:26:12.175 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 15 kB 00:00 2023-11-28T14:26:14.085 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-11-28T14:26:18.910 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-11-28T14:26:21.721 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:18.0.0-7503.g5537419c.el9 ceph 8.4 M 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.1 M 2023-11-28T14:26:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.0 M 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:18.0.0-7503.g5537419c.el9 ceph 6.6 M 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:18.0.0-7503.g5537419c.el9 ceph 17 M 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:18.0.0-7503.g5537419c.el9 ceph 24 k 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 728 k 2023-11-28T14:26:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 470 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:18.0.0-7503.g5537419c.el9 ceph 4.5 M 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: x86_64 2:18.0.0-7503.g5537419c.el9 ceph 45 k 2023-11-28T14:26:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:18.0.0-7503.g5537419c.el9 ceph 129 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:18.0.0-7503.g5537419c.el9 ceph 162 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:18.0.0-7503.g5537419c.el9 ceph 321 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 298 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:18.0.0-7503.g5537419c.el9 ceph 100 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:26:21.733 INFO:teuthology.orchestra.run.smithi038.stdout:Install 27 Packages 2023-11-28T14:26:21.734 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2023-11-28T14:26:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:21.734 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 54 M 2023-11-28T14:26:21.734 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:26:23.745 INFO:teuthology.orchestra.run.smithi038.stdout:(1/29): ceph-base-18.0.0-7503.g5537419c.el9.x86 3.4 MB/s | 6.6 MB 00:01 2023-11-28T14:26:23.812 INFO:teuthology.orchestra.run.smithi038.stdout:(2/29): ceph-selinux-18.0.0-7503.g5537419c.el9. 362 kB/s | 24 kB 00:00 2023-11-28T14:26:24.045 INFO:teuthology.orchestra.run.smithi038.stdout:(3/29): libcephfs2-18.0.0-7503.g5537419c.el9.x8 3.0 MB/s | 728 kB 00:00 2023-11-28T14:26:24.146 INFO:teuthology.orchestra.run.smithi038.stdout:(4/29): ceph-radosgw-18.0.0-7503.g5537419c.el9. 3.6 MB/s | 8.4 MB 00:02 2023-11-28T14:26:24.204 INFO:teuthology.orchestra.run.smithi038.stdout:(5/29): libradosstriper1-18.0.0-7503.g5537419c. 2.9 MB/s | 470 kB 00:00 2023-11-28T14:26:24.288 INFO:teuthology.orchestra.run.smithi038.stdout:(6/29): python3-ceph-argparse-18.0.0-7503.g5537 542 kB/s | 45 kB 00:00 2023-11-28T14:26:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:(7/29): python3-ceph-common-18.0.0-7503.g553741 1.3 MB/s | 129 kB 00:00 2023-11-28T14:26:24.471 INFO:teuthology.orchestra.run.smithi038.stdout:(8/29): python3-cephfs-18.0.0-7503.g5537419c.el 1.9 MB/s | 162 kB 00:00 2023-11-28T14:26:24.588 INFO:teuthology.orchestra.run.smithi038.stdout:(9/29): python3-rados-18.0.0-7503.g5537419c.el9 2.7 MB/s | 321 kB 00:00 2023-11-28T14:26:24.681 INFO:teuthology.orchestra.run.smithi038.stdout:(10/29): python3-rbd-18.0.0-7503.g5537419c.el9. 3.2 MB/s | 298 kB 00:00 2023-11-28T14:26:24.765 INFO:teuthology.orchestra.run.smithi038.stdout:(11/29): python3-rgw-18.0.0-7503.g5537419c.el9. 1.2 MB/s | 100 kB 00:00 2023-11-28T14:26:24.873 INFO:teuthology.orchestra.run.smithi038.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 980 kB/s | 106 kB 00:00 2023-11-28T14:26:24.915 INFO:teuthology.orchestra.run.smithi038.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2023-11-28T14:26:24.949 INFO:teuthology.orchestra.run.smithi038.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2023-11-28T14:26:24.999 INFO:teuthology.orchestra.run.smithi038.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-11-28T14:26:25.066 INFO:teuthology.orchestra.run.smithi038.stdout:(16/29): librgw2-18.0.0-7503.g5537419c.el9.x86_ 4.9 MB/s | 4.5 MB 00:00 2023-11-28T14:26:25.091 INFO:teuthology.orchestra.run.smithi038.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2023-11-28T14:26:25.108 INFO:teuthology.orchestra.run.smithi038.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2023-11-28T14:26:25.242 INFO:teuthology.orchestra.run.smithi038.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2023-11-28T14:26:25.275 INFO:teuthology.orchestra.run.smithi038.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 735 kB/s | 24 kB 00:00 2023-11-28T14:26:25.309 INFO:teuthology.orchestra.run.smithi038.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2023-11-28T14:26:25.351 INFO:teuthology.orchestra.run.smithi038.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2023-11-28T14:26:25.476 INFO:teuthology.orchestra.run.smithi038.stdout:(23/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 6.6 MB/s | 837 kB 00:00 2023-11-28T14:26:25.518 INFO:teuthology.orchestra.run.smithi038.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2023-11-28T14:26:25.619 INFO:teuthology.orchestra.run.smithi038.stdout:(25/29): libarrow-9.0.0-9.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2023-11-28T14:26:25.727 INFO:teuthology.orchestra.run.smithi038.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2023-11-28T14:26:26.511 INFO:teuthology.orchestra.run.smithi038.stdout:(27/29): librados2-18.0.0-7503.g5537419c.el9.x8 3.5 MB/s | 3.1 MB 00:00 2023-11-28T14:26:26.578 INFO:teuthology.orchestra.run.smithi038.stdout:(28/29): librbd1-18.0.0-7503.g5537419c.el9.x86_ 3.6 MB/s | 3.0 MB 00:00 2023-11-28T14:26:28.853 INFO:teuthology.orchestra.run.smithi038.stdout:(29/29): ceph-common-18.0.0-7503.g5537419c.el9. 2.4 MB/s | 17 MB 00:07 2023-11-28T14:26:28.856 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:26:28.856 INFO:teuthology.orchestra.run.smithi038.stdout:Total 7.5 MB/s | 54 MB 00:07 2023-11-28T14:26:28.857 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:26:28.898 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:26:28.898 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:26:29.376 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:26:29.376 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:26:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:26:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-28T14:26:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:26:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:26:31.422 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 3/31 2023-11-28T14:26:31.448 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:26:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:26:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-11-28T14:26:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-11-28T14:26:31.633 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-28T14:26:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9. 8/31 2023-11-28T14:26:31.821 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 9/31 2023-11-28T14:26:31.844 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 10/31 2023-11-28T14:26:31.916 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 10/31 2023-11-28T14:26:31.953 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-28T14:26:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-28T14:26:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-28T14:26:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-28T14:26:32.171 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-28T14:26:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-28T14:26:32.233 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-28T14:26:32.568 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-28T14:26:32.596 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 19/31 2023-11-28T14:26:32.687 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 19/31 2023-11-28T14:26:32.718 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 20/31 2023-11-28T14:26:32.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-28T14:26:33.215 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-28T14:26:33.249 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 23/31 2023-11-28T14:26:33.320 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 23/31 2023-11-28T14:26:33.397 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 24/31 2023-11-28T14:26:33.427 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x8 25/31 2023-11-28T14:26:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:26:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:26:36.320 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 26/31 2023-11-28T14:26:36.368 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 27/31 2023-11-28T14:26:36.436 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 27/31 2023-11-28T14:26:36.436 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T14:26:36.437 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T14:26:36.437 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:36.486 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:26:46.737 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:26:46.737 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2023-11-28T14:26:46.737 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2023-11-28T14:26:46.737 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2023-11-28T14:26:46.737 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2023-11-28T14:26:46.738 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2023-11-28T14:26:46.738 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2023-11-28T14:26:46.738 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2023-11-28T14:26:46.738 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:47.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 29/31 2023-11-28T14:26:47.637 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 29/31 2023-11-28T14:26:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:26:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T14:26:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T14:26:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T14:26:47.637 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:47.640 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-28T14:26:47.730 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-28T14:26:47.731 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 1/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 3/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 4/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 5/31 2023-11-28T14:26:48.824 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_6 6/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 7/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9. 8/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x8 9/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 10/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 11/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 12/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 13/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-28T14:26:48.825 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-28T14:26:48.826 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-28T14:26:48.826 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-28T14:26:48.826 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-28T14:26:48.826 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-28T14:26:48.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-28T14:26:48.829 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-28T14:26:48.830 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-28T14:26:48.830 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-28T14:26:48.830 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-28T14:26:48.830 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-28T14:26:48.830 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:18.0.0-7503.g5537419c.el9.x86_64 28/31 2023-11-28T14:26:48.830 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-28T14:26:48.831 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 30/31 2023-11-28T14:26:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T14:26:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T14:26:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:49.781 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:26:50.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2023-11-28T14:26:50.695 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:32 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:26:50.851 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:26:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:26:50.852 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:26:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:26:50.852 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:26:50.852 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:18.0.0-7503.g5537419c.el9 ceph 117 M 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:18.0.0-7503.g5537419c.el9 ceph 158 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:50.853 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:26:50.854 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:26:50.854 INFO:teuthology.orchestra.run.smithi038.stdout:Install 10 Packages 2023-11-28T14:26:50.854 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:26:50.854 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 120 M 2023-11-28T14:26:50.854 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.1 G 2023-11-28T14:26:50.855 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:26:51.084 INFO:teuthology.orchestra.run.smithi038.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-11-28T14:26:51.142 INFO:teuthology.orchestra.run.smithi038.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 7.2 MB/s | 418 kB 00:00 2023-11-28T14:26:51.184 INFO:teuthology.orchestra.run.smithi038.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-11-28T14:26:51.226 INFO:teuthology.orchestra.run.smithi038.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-11-28T14:26:51.268 INFO:teuthology.orchestra.run.smithi038.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-11-28T14:26:51.293 INFO:teuthology.orchestra.run.smithi038.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2023-11-28T14:26:51.377 INFO:teuthology.orchestra.run.smithi038.stdout:(7/10): libcephsqlite-18.0.0-7503.g5537419c.el9 340 kB/s | 158 kB 00:00 2023-11-28T14:26:51.652 INFO:teuthology.orchestra.run.smithi038.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 443 kB/s | 122 kB 00:00 2023-11-28T14:26:52.061 INFO:teuthology.orchestra.run.smithi038.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.6 MB/s | 1.3 MB 00:00 2023-11-28T14:26:56.711 INFO:teuthology.orchestra.run.smithi038.stdout:(10/10): ceph-test-18.0.0-7503.g5537419c.el9.x8 20 MB/s | 117 MB 00:05 2023-11-28T14:26:56.715 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:26:56.716 INFO:teuthology.orchestra.run.smithi038.stdout:Total 20 MB/s | 120 MB 00:05 2023-11-28T14:26:56.716 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:26:56.740 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:26:56.740 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:26:57.619 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:26:57.619 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:26:58.767 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:26:58.859 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-28T14:26:58.902 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-28T14:26:58.956 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-28T14:26:59.004 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-28T14:26:59.045 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-28T14:26:59.076 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-28T14:26:59.115 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-28T14:26:59.155 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-28T14:26:59.183 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 9/10 2023-11-28T14:27:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 9/10 2023-11-28T14:27:16.465 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 10/10 2023-11-28T14:27:24.808 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 10/10 2023-11-28T14:27:24.808 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 1/10 2023-11-28T14:27:24.808 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 2/10 2023-11-28T14:27:24.808 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-28T14:27:24.808 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-28T14:27:24.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-28T14:27:24.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-28T14:27:24.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-28T14:27:24.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-28T14:27:24.809 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: jq-1.6-16.el9.x86_64 2023-11-28T14:27:25.233 INFO:teuthology.orchestra.run.smithi038.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:27:25.234 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:27:25.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2023-11-28T14:27:26.140 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:08 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:27:26.309 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:27:26.311 INFO:teuthology.orchestra.run.smithi038.stdout:========================================================================================= 2023-11-28T14:27:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:27:26.311 INFO:teuthology.orchestra.run.smithi038.stdout:========================================================================================= 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:18.0.0-7503.g5537419c.el9 ceph 6.5 k 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:18.0.0-7503.g5537419c.el9 ceph 2.0 M 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:18.0.0-7503.g5537419c.el9 ceph 1.5 M 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 243 k 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:18.0.0-7503.g5537419c.el9 ceph 8.9 M 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 118 M 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T14:27:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T14:27:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T14:27:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout:========================================================================================= 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout:Install 33 Packages 2023-11-28T14:27:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:27:26.316 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 134 M 2023-11-28T14:27:26.316 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.4 G 2023-11-28T14:27:26.316 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:27:26.915 INFO:teuthology.orchestra.run.smithi038.stdout:(1/33): ceph-18.0.0-7503.g5537419c.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2023-11-28T14:27:27.023 INFO:teuthology.orchestra.run.smithi038.stdout:(2/33): ceph-mgr-18.0.0-7503.g5537419c.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2023-11-28T14:27:27.090 INFO:teuthology.orchestra.run.smithi038.stdout:(3/33): ceph-mds-18.0.0-7503.g5537419c.el9.x86_ 4.9 MB/s | 2.0 MB 00:00 2023-11-28T14:27:27.157 INFO:teuthology.orchestra.run.smithi038.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7503.g5537 3.6 MB/s | 243 kB 00:00 2023-11-28T14:27:27.266 INFO:teuthology.orchestra.run.smithi038.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2023-11-28T14:27:27.300 INFO:teuthology.orchestra.run.smithi038.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2023-11-28T14:27:27.333 INFO:teuthology.orchestra.run.smithi038.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-11-28T14:27:27.367 INFO:teuthology.orchestra.run.smithi038.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-28T14:27:27.409 INFO:teuthology.orchestra.run.smithi038.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-11-28T14:27:27.476 INFO:teuthology.orchestra.run.smithi038.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-11-28T14:27:27.509 INFO:teuthology.orchestra.run.smithi038.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-28T14:27:27.543 INFO:teuthology.orchestra.run.smithi038.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-11-28T14:27:27.577 INFO:teuthology.orchestra.run.smithi038.stdout:(13/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-28T14:27:27.610 INFO:teuthology.orchestra.run.smithi038.stdout:(14/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-11-28T14:27:27.894 INFO:teuthology.orchestra.run.smithi038.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2023-11-28T14:27:28.102 INFO:teuthology.orchestra.run.smithi038.stdout:(16/33): ceph-mon-18.0.0-7503.g5537419c.el9.x86 7.5 MB/s | 8.9 MB 00:01 2023-11-28T14:27:28.136 INFO:teuthology.orchestra.run.smithi038.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 710 kB/s | 172 kB 00:00 2023-11-28T14:27:28.211 INFO:teuthology.orchestra.run.smithi038.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2023-11-28T14:27:28.278 INFO:teuthology.orchestra.run.smithi038.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2023-11-28T14:27:28.345 INFO:teuthology.orchestra.run.smithi038.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 348 kB/s | 23 kB 00:00 2023-11-28T14:27:28.412 INFO:teuthology.orchestra.run.smithi038.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2023-11-28T14:27:28.480 INFO:teuthology.orchestra.run.smithi038.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2023-11-28T14:27:28.530 INFO:teuthology.orchestra.run.smithi038.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 840 kB/s | 358 kB 00:00 2023-11-28T14:27:28.555 INFO:teuthology.orchestra.run.smithi038.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2023-11-28T14:27:28.605 INFO:teuthology.orchestra.run.smithi038.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-11-28T14:27:28.680 INFO:teuthology.orchestra.run.smithi038.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-11-28T14:27:28.706 INFO:teuthology.orchestra.run.smithi038.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2023-11-28T14:27:28.765 INFO:teuthology.orchestra.run.smithi038.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2023-11-28T14:27:28.781 INFO:teuthology.orchestra.run.smithi038.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2023-11-28T14:27:28.873 INFO:teuthology.orchestra.run.smithi038.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2023-11-28T14:27:28.899 INFO:teuthology.orchestra.run.smithi038.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2023-11-28T14:27:28.950 INFO:teuthology.orchestra.run.smithi038.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2023-11-28T14:27:36.785 INFO:teuthology.orchestra.run.smithi038.stdout:(33/33): ceph-osd-18.0.0-7503.g5537419c.el9.x86 12 MB/s | 118 MB 00:09 2023-11-28T14:27:36.787 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:27:36.788 INFO:teuthology.orchestra.run.smithi038.stdout:Total 13 MB/s | 134 MB 00:10 2023-11-28T14:27:36.788 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:27:36.830 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:27:36.830 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:27:37.839 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:27:37.839 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:27:39.701 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:27:39.751 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-28T14:27:39.783 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-28T14:27:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-28T14:27:39.911 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-28T14:27:39.996 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-28T14:27:40.036 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-28T14:27:40.109 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-28T14:27:40.161 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-11-28T14:27:40.199 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-11-28T14:27:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-11-28T14:27:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-11-28T14:27:40.494 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-11-28T14:27:40.562 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-11-28T14:27:40.601 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-11-28T14:27:40.635 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-11-28T14:27:40.665 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-11-28T14:27:40.743 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-11-28T14:27:40.775 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-28T14:27:40.808 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-11-28T14:27:40.849 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-11-28T14:27:40.884 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-11-28T14:27:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-28T14:27:41.010 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-28T14:27:41.067 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-28T14:27:41.140 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-28T14:27:41.280 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-28T14:27:41.412 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-28T14:27:41.581 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9. 28/33 2023-11-28T14:27:41.612 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 29/33 2023-11-28T14:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 29/33 2023-11-28T14:27:41.639 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T14:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T14:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T14:27:41.640 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:01.028 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 30/33 2023-11-28T14:28:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 30/33 2023-11-28T14:28:01.056 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:01.056 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T14:28:01.056 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T14:28:01.056 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T14:28:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 31/33 2023-11-28T14:28:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 31/33 2023-11-28T14:28:02.022 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:02.022 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T14:28:02.022 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T14:28:02.022 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T14:28:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:02.238 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 32/33 2023-11-28T14:28:02.263 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 32/33 2023-11-28T14:28:02.263 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:02.263 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T14:28:02.263 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T14:28:02.263 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T14:28:02.264 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:02.329 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:18.0.0-7503.g5537419c.el9.x86_64 33/33 2023-11-28T14:28:12.620 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:18.0.0-7503.g5537419c.el9.x86_64 33/33 2023-11-28T14:28:12.620 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:18.0.0-7503.g5537419c.el9.x86_64 1/33 2023-11-28T14:28:12.620 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 2/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 3/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 4/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 5/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9. 6/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-11-28T14:28:12.621 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-28T14:28:12.622 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-28T14:28:12.623 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-28T14:28:12.623 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-28T14:28:12.623 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-28T14:28:12.623 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-28T14:28:12.624 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-28T14:28:12.624 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-28T14:28:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-28T14:28:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:13.563 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T14:28:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T14:28:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T14:28:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T14:28:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T14:28:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:13.568 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:13.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2023-11-28T14:28:14.405 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:56 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:14.502 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:28:14.555 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:14.556 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:28:14.556 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:14.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2023-11-28T14:28:15.217 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:57 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:15.368 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:15.369 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:15.369 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:15.369 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:15.369 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 775 k 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 775 k 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 789 k 2023-11-28T14:28:15.370 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:16.211 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-18.0.0-7503.g5537419c.el9.noarch.rpm 922 kB/s | 775 kB 00:00 2023-11-28T14:28:16.211 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:16.212 INFO:teuthology.orchestra.run.smithi038.stdout:Total 920 kB/s | 775 kB 00:00 2023-11-28T14:28:16.212 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:16.218 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:16.218 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:16.224 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:16.225 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:16.754 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:16.851 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:17.545 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:17.546 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:17.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2023-11-28T14:28:18.364 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:00 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:18.518 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:18.519 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:18.519 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2023-11-28T14:28:18.519 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:18.519 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:18.519 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7503.g5537419c.el9 ceph 143 k 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 143 k 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 437 k 2023-11-28T14:28:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:19.025 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-18.0.0-7503.g553741 284 kB/s | 143 kB 00:00 2023-11-28T14:28:19.026 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:19.026 INFO:teuthology.orchestra.run.smithi038.stdout:Total 283 kB/s | 143 kB 00:00 2023-11-28T14:28:19.026 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:19.034 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:19.034 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:19.091 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:19.091 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:19.236 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:19.309 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:28:19.336 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:28:19.336 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:19.336 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T14:28:19.336 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:20.349 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7503.g5537419c. 1/1 2023-11-28T14:28:20.349 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:20.349 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:20.350 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:20.350 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:20.350 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:20.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2023-11-28T14:28:21.165 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:03 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:21.263 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:28:21.316 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:21.317 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:28:21.318 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:21.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2023-11-28T14:28:21.974 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:03 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:22.128 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:22.129 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:22.129 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 1.8 M 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 28 k 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 15 k 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2023-11-28T14:28:22.130 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout:Install 7 Packages 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.1 M 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 8.7 M 2023-11-28T14:28:22.131 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:22.838 INFO:teuthology.orchestra.run.smithi038.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7503.g553 78 kB/s | 28 kB 00:00 2023-11-28T14:28:22.871 INFO:teuthology.orchestra.run.smithi038.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7503.g5537 38 kB/s | 15 kB 00:00 2023-11-28T14:28:22.988 INFO:teuthology.orchestra.run.smithi038.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 60 kB/s | 9.0 kB 00:00 2023-11-28T14:28:23.014 INFO:teuthology.orchestra.run.smithi038.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 350 kB/s | 41 kB 00:00 2023-11-28T14:28:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 230 kB/s | 31 kB 00:00 2023-11-28T14:28:23.172 INFO:teuthology.orchestra.run.smithi038.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.4 MB/s | 188 kB 00:00 2023-11-28T14:28:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7503.g5537419c 989 kB/s | 1.8 MB 00:01 2023-11-28T14:28:24.300 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:24.301 INFO:teuthology.orchestra.run.smithi038.stdout:Total 973 kB/s | 2.1 MB 00:02 2023-11-28T14:28:24.301 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:24.822 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:24.901 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-28T14:28:24.933 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-28T14:28:24.989 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-28T14:28:25.019 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-28T14:28:25.050 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.n 5/7 2023-11-28T14:28:25.300 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9. 6/7 2023-11-28T14:28:25.335 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 7/7 2023-11-28T14:28:25.872 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 7/7 2023-11-28T14:28:25.873 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9. 1/7 2023-11-28T14:28:25.873 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarc 2/7 2023-11-28T14:28:25.873 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.n 3/7 2023-11-28T14:28:25.873 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-28T14:28:25.873 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-28T14:28:25.874 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-28T14:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-28T14:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T14:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T14:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T14:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T14:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:26.311 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:26.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-28T14:28:27.126 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:09 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:27.279 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:27.280 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2023-11-28T14:28:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:27.280 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2023-11-28T14:28:27.280 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 7.4 M 2023-11-28T14:28:27.280 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T14:28:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:27.282 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:27.282 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2023-11-28T14:28:27.282 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2023-11-28T14:28:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:27.282 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 37 M 2023-11-28T14:28:27.282 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 209 M 2023-11-28T14:28:27.283 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:27.704 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2023-11-28T14:28:27.737 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2023-11-28T14:28:27.771 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-11-28T14:28:27.838 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-11-28T14:28:27.955 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-11-28T14:28:28.047 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2023-11-28T14:28:28.072 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-11-28T14:28:28.249 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2023-11-28T14:28:28.415 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2023-11-28T14:28:28.440 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-11-28T14:28:28.590 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2023-11-28T14:28:29.749 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 3.4 MB/s | 7.4 MB 00:02 2023-11-28T14:28:29.752 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:29.752 INFO:teuthology.orchestra.run.smithi038.stdout:Total 15 MB/s | 37 MB 00:02 2023-11-28T14:28:29.767 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:29.795 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:29.796 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:30.250 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:30.251 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-28T14:28:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-28T14:28:31.410 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-28T14:28:31.457 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-28T14:28:31.645 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-28T14:28:31.672 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-28T14:28:31.761 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-28T14:28:32.580 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-28T14:28:32.700 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-28T14:28:34.004 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-28T14:28:35.106 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-28T14:28:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 12/12 2023-11-28T14:28:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 12/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7503.g55374 1/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-28T14:28:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T14:28:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:38.488 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:38.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2023-11-28T14:28:39.437 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:21 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:39.591 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 50 k 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T14:28:39.592 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T14:28:39.593 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:39.594 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:39.594 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2023-11-28T14:28:39.594 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2023-11-28T14:28:39.594 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:39.594 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.0 M 2023-11-28T14:28:39.595 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2023-11-28T14:28:39.595 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:39.774 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 141 kB/s | 18 kB 00:00 2023-11-28T14:28:39.799 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 159 kB/s | 24 kB 00:00 2023-11-28T14:28:39.849 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-11-28T14:28:39.882 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2023-11-28T14:28:39.908 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-11-28T14:28:39.933 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-11-28T14:28:40.008 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2023-11-28T14:28:40.034 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): ceph-mgr-rook-18.0.0-7503.g5537419c.el9 129 kB/s | 50 kB 00:00 2023-11-28T14:28:40.059 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2023-11-28T14:28:40.276 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 739 kB/s | 197 kB 00:00 2023-11-28T14:28:40.301 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2023-11-28T14:28:40.326 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2023-11-28T14:28:40.376 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2023-11-28T14:28:40.378 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:40.378 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-11-28T14:28:40.379 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:40.394 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:40.394 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:40.501 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:40.502 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:41.039 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:41.217 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-28T14:28:41.346 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-28T14:28:41.402 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-28T14:28:41.446 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-28T14:28:41.488 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-28T14:28:41.585 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-28T14:28:41.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-28T14:28:41.757 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-28T14:28:42.528 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-28T14:28:42.578 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-28T14:28:42.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-28T14:28:42.660 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-28T14:28:42.689 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 13/13 2023-11-28T14:28:43.485 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 13/13 2023-11-28T14:28:43.485 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 1/13 2023-11-28T14:28:43.485 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-28T14:28:43.485 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-28T14:28:43.486 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T14:28:43.857 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:43.858 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:44.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2023-11-28T14:28:44.720 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:26 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:44.873 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 133 k 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T14:28:44.874 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2023-11-28T14:28:44.875 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:44.876 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 6.8 M 2023-11-28T14:28:44.876 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2023-11-28T14:28:44.876 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:45.353 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-18.0.0-7503.g5537419c.e 528 kB/s | 133 kB 00:00 2023-11-28T14:28:50.738 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 607 kB/s | 228 kB 00:00 2023-11-28T14:28:50.738 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.2 MB/s | 5.8 MB 00:00 2023-11-28T14:28:50.738 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 136 kB/s | 58 kB 00:00 2023-11-28T14:28:50.739 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 130 kB/s | 51 kB 00:00 2023-11-28T14:28:50.739 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 626 kB/s | 548 kB 00:00 2023-11-28T14:28:50.739 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:50.739 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2023-11-28T14:28:50.761 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:50.761 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:50.761 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:50.761 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:50.761 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:50.762 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:50.762 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-28T14:28:50.762 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-28T14:28:50.762 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-28T14:28:50.762 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 6/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 6/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 1/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-28T14:28:50.763 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-28T14:28:50.764 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-28T14:28:50.764 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-28T14:28:50.764 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:50.764 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:50.765 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:50.765 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T14:28:50.765 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T14:28:50.765 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T14:28:50.791 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T14:28:50.791 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T14:28:50.792 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:50.792 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:50.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2023-11-28T14:28:51.410 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:33 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:51.560 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:51.561 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:51.561 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:51.561 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:18.0.0-7503.g5537419c.el9 ceph 832 k 2023-11-28T14:28:51.561 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:51.562 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 911 k 2023-11-28T14:28:51.563 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 3.0 M 2023-11-28T14:28:51.563 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:51.681 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 664 kB/s | 79 kB 00:00 2023-11-28T14:28:51.873 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-18.0.0-7503.g5537419c.el9.x86_ 2.6 MB/s | 832 kB 00:00 2023-11-28T14:28:51.874 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:51.875 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.8 MB/s | 911 kB 00:00 2023-11-28T14:28:51.875 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:51.897 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:51.978 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:51.978 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:52.182 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:52.318 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-28T14:28:52.400 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2/2 2023-11-28T14:28:53.049 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2/2 2023-11-28T14:28:53.050 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/2 2023-11-28T14:28:53.379 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-28T14:28:53.379 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:53.379 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-28T14:28:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:53.380 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:53.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-volume 2023-11-28T14:28:54.259 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:36 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:54.410 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:54.411 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:54.411 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:28:54.411 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:18.0.0-7503.g5537419c.el9 ceph-noarch 262 k 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 262 k 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.2 M 2023-11-28T14:28:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:55.169 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-volume-18.0.0-7503.g5537419c.el9.noarch.rp 346 kB/s | 262 kB 00:00 2023-11-28T14:28:55.170 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:55.170 INFO:teuthology.orchestra.run.smithi038.stdout:Total 346 kB/s | 262 kB 00:00 2023-11-28T14:28:55.170 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:55.179 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:55.179 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:55.214 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:55.215 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:55.440 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:55.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:55.546 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:28:55.546 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T14:28:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 1/1 2023-11-28T14:28:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:56.572 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:18.0.0-7503.g5537419c.el9.noarch 2023-11-28T14:28:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:56.572 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:56.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2023-11-28T14:28:57.509 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:39 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:28:57.661 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 127 k 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:28:57.662 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:57.663 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 127 k 2023-11-28T14:28:57.663 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 456 k 2023-11-28T14:28:57.663 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:28:58.199 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-18.0.0-7503.g5537419c.el9.x86_64 236 kB/s | 127 kB 00:00 2023-11-28T14:28:58.200 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:28:58.200 INFO:teuthology.orchestra.run.smithi038.stdout:Total 235 kB/s | 127 kB 00:00 2023-11-28T14:28:58.200 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:28:58.206 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:28:58.207 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:28:58.266 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:28:58.267 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:28:58.429 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:28:58.542 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:28:58.978 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:28:59.316 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:28:59.316 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:59.317 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:28:59.317 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:28:59.317 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:28:59.317 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:28:59.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2023-11-28T14:29:00.136 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:42 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:00.234 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:00.288 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:00.290 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:00.290 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:00.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2023-11-28T14:29:00.956 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:42 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:01.109 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:01.109 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 31 k 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 31 k 2023-11-28T14:29:01.110 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 139 k 2023-11-28T14:29:01.111 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:29:01.612 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-18.0.0-7503.g5537419c.el9.x86_6 62 kB/s | 31 kB 00:00 2023-11-28T14:29:01.612 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:29:01.613 INFO:teuthology.orchestra.run.smithi038.stdout:Total 62 kB/s | 31 kB 00:00 2023-11-28T14:29:01.613 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:29:01.616 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:29:01.617 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:29:01.651 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:29:01.652 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:29:01.744 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:29:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:02.270 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:02.560 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:29:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:29:02.560 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:02.561 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:02.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2023-11-28T14:29:03.337 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:45 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:03.434 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:03.488 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:03.489 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:03.489 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:03.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2023-11-28T14:29:04.156 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:46 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:04.252 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:04.305 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:04.306 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:04.306 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:04.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2023-11-28T14:29:04.968 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:46 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:05.068 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:05.123 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:05.124 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:05.124 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:05.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2023-11-28T14:29:05.803 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:47 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:05.907 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:05.963 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:05.964 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:05.964 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:06.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2023-11-28T14:29:06.670 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:48 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:06.770 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:06.824 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:06.826 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:06.826 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:06.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2023-11-28T14:29:07.481 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:49 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:07.581 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:18.0.0-7503.g5537419c.el9.x86_64 is already installed. 2023-11-28T14:29:07.635 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:07.636 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-11-28T14:29:07.637 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:07.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2023-11-28T14:29:08.314 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:50 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:08.464 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:08.465 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:08.465 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-28T14:29:08.465 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:08.465 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:29:08.465 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:18.0.0-7503.g5537419c.el9 ceph 86 k 2023-11-28T14:29:08.465 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 86 k 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 256 k 2023-11-28T14:29:08.466 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:29:09.032 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-18.0.0-7503.g5537419c.el9.x86_64.rpm 151 kB/s | 86 kB 00:00 2023-11-28T14:29:09.033 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:29:09.033 INFO:teuthology.orchestra.run.smithi038.stdout:Total 151 kB/s | 86 kB 00:00 2023-11-28T14:29:09.033 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:29:09.041 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:29:09.041 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:29:09.101 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:29:09.101 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:29:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:29:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:10.049 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:10.049 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:10.049 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:29:10.049 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:29:10.049 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:10.049 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:10.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2023-11-28T14:29:10.874 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:52 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:11.029 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:11.030 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:11.030 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2023-11-28T14:29:11.030 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:11.030 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:29:11.030 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:18.0.0-7503.g5537419c.el9 ceph 3.0 M 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.0 M 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 13 M 2023-11-28T14:29:11.031 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:29:12.119 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-18.0.0-7503.g5537419c.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2023-11-28T14:29:12.121 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:29:12.121 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.7 MB/s | 3.0 MB 00:01 2023-11-28T14:29:12.121 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:29:12.131 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:29:12.132 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:29:12.210 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:29:12.211 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:29:12.672 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:29:12.728 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:12.754 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T14:29:12.754 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T14:29:12.755 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T14:29:12.755 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T14:29:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:13.936 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:29:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:29:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:13.937 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:14.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2023-11-28T14:29:14.782 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:02:56 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:29:14.933 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:18.0.0-7503.g5537419c.el9 ceph 169 k 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:14.934 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 169 k 2023-11-28T14:29:14.935 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 536 k 2023-11-28T14:29:14.935 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:29:15.622 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-18.0.0-7503.g5537419c.el9.x86_64.rpm 246 kB/s | 169 kB 00:00 2023-11-28T14:29:15.623 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:29:15.623 INFO:teuthology.orchestra.run.smithi038.stdout:Total 245 kB/s | 169 kB 00:00 2023-11-28T14:29:15.623 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:29:15.632 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:29:15.632 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:29:15.693 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:29:15.693 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:29:15.848 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:29:15.957 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:16.720 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 1/1 2023-11-28T14:29:16.720 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:16.721 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:29:16.721 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:29:16.721 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:16.721 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:29:16.943 DEBUG:teuthology.parallel:result is None 2023-11-28T14:29:16.943 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=5537419ca37a48f0ea28d7872ae1f14ccf1a953b 2023-11-28T14:29:17.159 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T14:29:17.182 INFO:teuthology.orchestra.run.smithi038.stdout:18.0.0-7503.g5537419c.el9 2023-11-28T14:29:17.183 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7503.g5537419c.el9 2023-11-28T14:29:17.183 INFO:teuthology.task.install:The correct ceph version 18.0.0-7503.g5537419c is installed. 2023-11-28T14:29:17.183 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T14:29:17.183 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:17.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T14:29:17.255 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T14:29:17.255 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:17.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T14:29:17.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T14:29:17.398 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T14:29:17.398 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:17.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T14:29:17.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T14:29:17.540 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T14:29:17.540 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:17.540 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T14:29:17.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T14:29:17.682 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T14:29:17.746 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T14:29:17.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2023-11-28T14:29:17.776 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T14:29:17.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T14:29:17.847 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T14:29:17.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T14:29:17.926 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T14:29:17.926 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T14:29:17.926 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175', 'branch': 'wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read}', 'email': None, 'first_in_suite': False, 'job_id': '7470175', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'b9b6c42eed672399c3d01aa72f6cda1a665a3c35'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9581, 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_b9b6c42eed672399c3d01aa72f6cda1a665a3c35/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'b9b6c42eed672399c3d01aa72f6cda1a665a3c35', 'targets': {'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBTRdcTUMH9vNDqA18WeFcFNIb7sxkBy1gegmQd57B94t5Nr/JXr4SFs5KEqqXIDvjAFecctn0ydU67mzIyR+r4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5537419ca37a48f0ea28d7872ae1f14ccf1a953b', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_14:04:23', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T14:29:17.927 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T14:29:17.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T14:29:18.044 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:18.044 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T14:29:18.100 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T14:29:18.100 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2023-11-28T14:29:18.155 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-11-28 14:29:12.987858730 +0000 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-11-28 14:25:22.135490394 +0000 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-11-28 14:25:22.135490394 +0000 2023-11-28T14:29:18.156 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2023-11-28 14:25:22.135490394 +0000 2023-11-28T14:29:18.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T14:29:18.224 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-11-28T14:29:18.224 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-11-28T14:29:18.224 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000296434 s, 1.7 MB/s 2023-11-28T14:29:18.225 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T14:29:18.281 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2023-11-28T14:29:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T14:29:18.338 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:29:18.338 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-11-28T14:29:18.339 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:29:18.339 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:29:18.339 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-11-28 14:29:12.988858710 +0000 2023-11-28T14:29:18.339 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-11-28 14:25:22.693479580 +0000 2023-11-28T14:29:18.339 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-11-28 14:25:22.693479580 +0000 2023-11-28T14:29:18.339 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2023-11-28 14:25:22.693479580 +0000 2023-11-28T14:29:18.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T14:29:18.407 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-11-28T14:29:18.407 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-11-28T14:29:18.408 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000233277 s, 2.2 MB/s 2023-11-28T14:29:18.408 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T14:29:18.464 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2023-11-28T14:29:18.519 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T14:29:18.519 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-11-28 14:29:12.988858710 +0000 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-11-28 14:25:23.215469464 +0000 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-11-28 14:25:23.215469464 +0000 2023-11-28T14:29:18.520 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2023-11-28 14:25:23.215469464 +0000 2023-11-28T14:29:18.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T14:29:18.588 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-11-28T14:29:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-11-28T14:29:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000279315 s, 1.8 MB/s 2023-11-28T14:29:18.590 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T14:29:18.646 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2023-11-28T14:29:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-11-28 14:29:12.988858710 +0000 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-11-28 14:25:23.750459097 +0000 2023-11-28T14:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-11-28 14:25:23.750459097 +0000 2023-11-28T14:29:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2023-11-28 14:25:23.750459097 +0000 2023-11-28T14:29:18.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T14:29:18.770 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-11-28T14:29:18.770 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-11-28T14:29:18.770 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.0002341 s, 2.2 MB/s 2023-11-28T14:29:18.771 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T14:29:18.827 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-28T14:29:18.827 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-28T14:29:18.827 INFO:tasks.ceph:Generating config... 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-28T14:29:18.829 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-28T14:29:18.830 INFO:tasks.ceph:[osd] log to file = False 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T14:29:18.831 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-28T14:29:18.831 INFO:tasks.ceph:[global] auth client required = none 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] auth cluster required = none 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] auth service required = none 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] auth supported = none 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug asok = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug auth = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug client = 0/0 2023-11-28T14:29:18.832 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug context = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug crush = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug filer = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug journal = 0/0 2023-11-28T14:29:18.833 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mds = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-11-28T14:29:18.834 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug mon = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug monc = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug ms = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug osd = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug rados = 0/0 2023-11-28T14:29:18.835 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug refs = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug striper = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug timer = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] debug tp = 0/0 2023-11-28T14:29:18.836 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-28T14:29:18.837 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-28T14:29:18.837 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-28T14:29:18.837 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-28T14:29:18.837 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-28T14:29:18.837 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-28T14:29:18.837 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-28T14:29:18.837 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T14:29:18.837 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T14:29:18.837 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-11-28T14:29:18.838 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-11-28T14:29:18.838 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-11-28T14:29:18.838 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-28T14:29:18.838 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T14:29:18.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T14:29:18.925 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T14:29:18.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T14:29:18.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T14:29:19.012 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.38')] 2023-11-28T14:29:19.013 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-28T14:29:19.014 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:19.014 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T14:29:19.068 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:19.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:19.152+0000 7f44aaf44e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:19.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:19.152+0000 7f44aaf44e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:19.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:19.152+0000 7f44aaf44e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:19.155 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:19.155 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid e230aabf-8481-4c61-a863-2290fb6e5a4e 2023-11-28T14:29:19.155 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = pacific 2023-11-28T14:29:19.155 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2023-11-28T14:29:19.155 INFO:teuthology.orchestra.run.smithi038.stdout:fsid e230aabf-8481-4c61-a863-2290fb6e5a4e 2023-11-28T14:29:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2023-11-28T14:29:19.154355+0000 2023-11-28T14:29:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:created 2023-11-28T14:29:19.154355+0000 2023-11-28T14:29:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 16 (pacific) 2023-11-28T14:29:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 1 2023-11-28T14:29:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2023-11-28T14:29:19.156 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-28T14:29:19.158 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T14:29:19.173 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e230aabf-8481-4c61-a863-2290fb6e5a4e... 2023-11-28T14:29:19.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T14:29:19.274 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2023-11-28T14:29:19.274 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = none 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2023-11-28T14:29:19.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.276 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T14:29:19.277 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = 172.21.15.38 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: auth client required = none 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: auth cluster required = none 2023-11-28T14:29:19.278 INFO:teuthology.orchestra.run.smithi038.stdout: auth service required = none 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug asok = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug bdev = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug buffer = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug civetweb = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug client = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug compressor = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug context = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug crush = 0/0 2023-11-28T14:29:19.279 INFO:teuthology.orchestra.run.smithi038.stdout: debug crypto = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug dpdk = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug eventtrace = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug filer = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug filestore = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug finisher = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug fuse = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug heartbeatmap = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug javaclient = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug journal = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug journaler = 0/0 2023-11-28T14:29:19.280 INFO:teuthology.orchestra.run.smithi038.stdout: debug kstore = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug leveldb = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug lockdep = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mds = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mds balancer = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mds locker = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mds log = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mds log expire = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mds migrator = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug memdb = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgrc = 0/0 2023-11-28T14:29:19.281 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug monc = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug objclass = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug objectcacher = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug objecter = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug optracker = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug perfcounter = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug rados = 0/0 2023-11-28T14:29:19.282 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd mirror = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug rbd replay = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug refs = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug rgw = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug rgw sync = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug striper = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug throttle = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug timer = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: debug tp = 0/0 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-28T14:29:19.283 INFO:teuthology.orchestra.run.smithi038.stdout: ms client mode = crc 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: ms cluster mode = crc 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: ms mon client mode = crc 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: ms mon cluster mode = crc 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: ms mon service mode = crc 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: ms service mode = crc 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = e230aabf-8481-4c61-a863-2290fb6e5a4e 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.284 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 600 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock profile = high_recovery_ops 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug trim objects = true 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.285 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.286 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: debug monc = 20 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression algorithm = snappy 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression mode = aggressive 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore rocksdb cf = False 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: crimson seastar smp = 3 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 20 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 20 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: debug filestore = 0/0 2023-11-28T14:29:19.287 INFO:teuthology.orchestra.run.smithi038.stdout: debug journal = 0/0 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 0/0 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 0/0 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 10 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: err to stderr = True 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: log flush on exit = True 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: log to file = False 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: log to stderr = True 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2023-11-28T14:29:19.288 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd shutdown pgref assert = True 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.289 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 0/0 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 0/0 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 0/0 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.290 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: # 1m isn't quite enough 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: mon_down_mkfs_grace = 2m 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.291 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_filestore_osds = false 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default crimson = True 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T14:29:19.292 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2023-11-28T14:29:19.293 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T14:29:19.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T14:29:19.399 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T14:29:19.399 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:19.399 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T14:29:19.416 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:19.416 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T14:29:19.470 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:29:19.470 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:19.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T14:29:19.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T14:29:19.557 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:19.557 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:19.613 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T14:29:19.613 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T14:29:19.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T14:29:19.723 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T14:29:19.724 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T14:29:19.724 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T14:29:19.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T14:29:19.775 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T14:29:19.794 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T14:29:19.795 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-28T14:29:19.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T14:29:19.825 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-28T14:29:19.825 INFO:tasks.ceph:role: osd.0 2023-11-28T14:29:19.825 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:29:19.826 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T14:29:19.923 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:29:19.923 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:29:19.923 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:29:19.924 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:29:21.240 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2023-11-28T14:29:21.242 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2023-11-28T14:29:21.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T14:29:21.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T14:29:21.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T14:29:21.441 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-28T14:29:21.492 INFO:tasks.ceph:role: osd.1 2023-11-28T14:29:21.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:29:21.493 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T14:29:21.556 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:29:21.556 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:29:21.556 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:29:21.556 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:29:21.556 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:29:21.557 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:29:21.557 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:29:21.557 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:29:21.557 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:29:21.557 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:29:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2023-11-28T14:29:22.879 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2023-11-28T14:29:22.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T14:29:22.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T14:29:22.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T14:29:23.065 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-28T14:29:23.065 INFO:tasks.ceph:role: osd.2 2023-11-28T14:29:23.065 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:29:23.065 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T14:29:23.167 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T14:29:24.484 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2023-11-28T14:29:24.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2023-11-28T14:29:24.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-28T14:29:24.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T14:29:24.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:29:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,160 seastar - Reactor backend: linux-aio 2023-11-28T14:29:25.176 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,175 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:29:25.176 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,175 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:29:25.176 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,175 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:29:25.176 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,175 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:29:25.178 INFO:teuthology.orchestra.run.smithi038.stderr:==80918==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:29:25.184 INFO:teuthology.orchestra.run.smithi038.stderr:==80918==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc9b5f4000; bottom 0x7fe1013d7000; size: 0x001b9a21d000 (118550024192) 2023-11-28T14:29:25.184 INFO:teuthology.orchestra.run.smithi038.stderr:False positive error reports may follow 2023-11-28T14:29:25.184 INFO:teuthology.orchestra.run.smithi038.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:29:25.210 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x2c275 0x2c5a7 0x2857c 0xb6323 0x22ea3063 0x22ea16b4 0x22ea16b4 0x2316aada 0x2316cf1e 0x2316da49 0x22fd3529 0x2309d264 0x2309ecaf 0x2309fa18 0x23067884 0x23066edc 0x22d7810c 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:29:25.210 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:25.725 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:25.726 INFO:teuthology.orchestra.run.smithi038.stderr:Suppressions used: 2023-11-28T14:29:25.726 INFO:teuthology.orchestra.run.smithi038.stderr: count bytes template 2023-11-28T14:29:25.726 INFO:teuthology.orchestra.run.smithi038.stderr: 1 8 MallocExtension::Register 2023-11-28T14:29:25.726 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:25.726 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:29:25.785 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:29:25.785 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,785 seastar - Reactor backend: linux-aio 2023-11-28T14:29:25.803 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,803 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:29:25.803 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,803 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:29:25.804 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,803 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:29:25.811 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:25,811 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:29:25.813 INFO:teuthology.orchestra.run.smithi038.stderr:==80916==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:29:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:==80916==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc9b5f4000; bottom 0x7fe0ff1d0000; size: 0x001b9c424000 (118585704448) 2023-11-28T14:29:25.819 INFO:teuthology.orchestra.run.smithi038.stderr:False positive error reports may follow 2023-11-28T14:29:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:29:25.845 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x22dab196 0x22ed2df3 0x24db534b 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1ee31d3a 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:29:25.846 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:25.906 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26619c5e 0x2661c2a7 0x2661e0b8 0x2662091d 0x26446cb1 0x1eeea284 0x1ee5ffaf 0x1ee63df0 0x2ce06e05 2023-11-28T14:29:25.906 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff813508d3 0xffffffff8135091a 0xffffffff81e6146c 0xffffffff820000ea 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stderr:Suppressions used: 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stderr: count bytes template 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stderr: 1 8 MallocExtension::Register 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:29:29.874 INFO:teuthology.orchestra.run.smithi038.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid e230aabf-8481-4c61-a863-2290fb6e5a4e 2023-11-28T14:29:29.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T14:29:29.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:30.500 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:29:30.501 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:30,501 seastar - Reactor backend: linux-aio 2023-11-28T14:29:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:30,516 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:29:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:30,516 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:29:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:30,516 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:29:30.517 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:30,517 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:29:30.519 INFO:teuthology.orchestra.run.smithi038.stderr:==81100==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:29:30.525 INFO:teuthology.orchestra.run.smithi038.stderr:==81100==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc53298000; bottom 0x7fcdfd2d4000; size: 0x002e55fc4000 (199011090432) 2023-11-28T14:29:30.526 INFO:teuthology.orchestra.run.smithi038.stderr:False positive error reports may follow 2023-11-28T14:29:30.526 INFO:teuthology.orchestra.run.smithi038.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:29:30.551 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x23076919 0x23080862 0x230d0144 0x230944e5 0x23097b0e 0x230abaff 0x230cb7a1 0x230cc029 0x23079bf9 0x23089219 0x2308a915 0x2308b2ee 0x2308c3d9 0x2316e328 0x22fd3529 0x2309d264 0x2309ecaf 0x2309fa18 0x23067884 0x23066edc 0x22d7810c 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:29:30.551 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:31.062 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:31.062 INFO:teuthology.orchestra.run.smithi038.stderr:Suppressions used: 2023-11-28T14:29:31.062 INFO:teuthology.orchestra.run.smithi038.stderr: count bytes template 2023-11-28T14:29:31.062 INFO:teuthology.orchestra.run.smithi038.stderr: 1 8 MallocExtension::Register 2023-11-28T14:29:31.063 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:31.063 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:29:31.118 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:29:31.118 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:31,118 seastar - Reactor backend: linux-aio 2023-11-28T14:29:31.136 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:31,136 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:29:31.137 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:31,136 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:29:31.137 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:31,136 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:29:31.144 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:31,144 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:29:31.146 INFO:teuthology.orchestra.run.smithi038.stderr:==81098==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:29:31.152 INFO:teuthology.orchestra.run.smithi038.stderr:==81098==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc53298000; bottom 0x7fcdfb0cd000; size: 0x002e581cb000 (199046770688) 2023-11-28T14:29:31.153 INFO:teuthology.orchestra.run.smithi038.stderr:False positive error reports may follow 2023-11-28T14:29:31.153 INFO:teuthology.orchestra.run.smithi038.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:29:31.178 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x1eba33ac 0x1edf07d0 0x1edf0bfa 0x1ed6afb5 0x22ecd56c 0x22ed2af8 0x24db534b 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1ee31d3a 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:29:31.179 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:31.241 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26619c5e 0x2661c2a7 0x2661e0b8 0x2662091d 0x26446cb1 0x1eeea284 0x1ee5ffaf 0x1ee63df0 0x2ce06e05 2023-11-28T14:29:31.241 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff813508d3 0xffffffff8135091a 0xffffffff81e6146c 0xffffffff820000ea 2023-11-28T14:29:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:Suppressions used: 2023-11-28T14:29:35.231 INFO:teuthology.orchestra.run.smithi038.stderr: count bytes template 2023-11-28T14:29:35.232 INFO:teuthology.orchestra.run.smithi038.stderr: 1 8 MallocExtension::Register 2023-11-28T14:29:35.232 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:35.232 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:29:35.232 INFO:teuthology.orchestra.run.smithi038.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid e230aabf-8481-4c61-a863-2290fb6e5a4e 2023-11-28T14:29:35.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T14:29:35.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:35.856 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:29:35.857 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:35,857 seastar - Reactor backend: linux-aio 2023-11-28T14:29:35.873 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:35,872 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:29:35.873 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:35,872 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:29:35.874 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:35,872 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:29:35.874 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:35,873 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:29:35.875 INFO:teuthology.orchestra.run.smithi038.stderr:==81282==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:29:35.881 INFO:teuthology.orchestra.run.smithi038.stderr:==81282==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff0e834000; bottom 0x7f4599c9d000; size: 0x00b974b97000 (796527259648) 2023-11-28T14:29:35.882 INFO:teuthology.orchestra.run.smithi038.stderr:False positive error reports may follow 2023-11-28T14:29:35.882 INFO:teuthology.orchestra.run.smithi038.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:29:35.907 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x22fe58ab 0x2306ca86 0x2306db40 0x2307dcdb 0x23080d03 0x230d0144 0x230944e5 0x23097b0e 0x230abaff 0x230cb7a1 0x230cc029 0x23079bf9 0x23089219 0x2308a915 0x2308b2ee 0x2308c3d9 0x2316e328 0x22fd3529 0x2309d264 0x2309ecaf 0x2309fa18 0x23067884 0x23066edc 0x22d7810c 0x24db4880 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1efbb1a0 0x1efc3f3d 0x1ee30484 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:29:35.907 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:36.418 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:36.418 INFO:teuthology.orchestra.run.smithi038.stderr:Suppressions used: 2023-11-28T14:29:36.418 INFO:teuthology.orchestra.run.smithi038.stderr: count bytes template 2023-11-28T14:29:36.418 INFO:teuthology.orchestra.run.smithi038.stderr: 1 8 MallocExtension::Register 2023-11-28T14:29:36.418 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:36.419 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:29:36.475 INFO:teuthology.orchestra.run.smithi038.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-28T14:29:36.475 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:36,475 seastar - Reactor backend: linux-aio 2023-11-28T14:29:36.503 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:36,503 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-28T14:29:36.503 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:36,503 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-28T14:29:36.503 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:36,503 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-28T14:29:36.510 INFO:teuthology.orchestra.run.smithi038.stderr:INFO 2023-11-28 14:29:36,510 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-28T14:29:36.513 INFO:teuthology.orchestra.run.smithi038.stderr:==81280==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-28T14:29:36.519 INFO:teuthology.orchestra.run.smithi038.stderr:==81280==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff0e834000; bottom 0x7f4597a96000; size: 0x00b976d9e000 (796562939904) 2023-11-28T14:29:36.520 INFO:teuthology.orchestra.run.smithi038.stderr:False positive error reports may follow 2023-11-28T14:29:36.520 INFO:teuthology.orchestra.run.smithi038.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-28T14:29:36.545 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0x22dc99b0 0x22e1eefd 0x22e3e8c8 0x22ec8321 0x22ecd487 0x22ed2af8 0x24db534b 0x24dbc036 0x24dbe984 0x24dbfb5d 0x2291858d 0x229691ea 0x2298cf1f 0x2ca25337 0x2cac3152 0x2cce73b0 0x2cce90d3 0x2c73e9fa 0x2c740903 0x1ee31d3a 0x3feaf 0x3ff5f 0x1e8daa44 2023-11-28T14:29:36.545 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:36.608 INFO:teuthology.orchestra.run.smithi038.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2ca8301c 0x2ca843ca 0x2ca85918 0x2ca85c9c 0x2ca85de6 0x2ca8623c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26619c5e 0x2661c2a7 0x2661e0b8 0x2662091d 0x26446cb1 0x1eeea284 0x1ee5ffaf 0x1ee63df0 0x2ce06e05 2023-11-28T14:29:36.608 INFO:teuthology.orchestra.run.smithi038.stderr:kernel callstack: 2023-11-28T14:29:40.555 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:40.556 INFO:teuthology.orchestra.run.smithi038.stderr:Suppressions used: 2023-11-28T14:29:40.556 INFO:teuthology.orchestra.run.smithi038.stderr: count bytes template 2023-11-28T14:29:40.556 INFO:teuthology.orchestra.run.smithi038.stderr: 1 8 MallocExtension::Register 2023-11-28T14:29:40.556 INFO:teuthology.orchestra.run.smithi038.stderr:----------------------------------------------------- 2023-11-28T14:29:40.556 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:29:40.556 INFO:teuthology.orchestra.run.smithi038.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid e230aabf-8481-4c61-a863-2290fb6e5a4e 2023-11-28T14:29:40.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T14:29:40.626 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T14:29:40.626 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:40.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T14:29:40.697 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:40.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T14:29:40.768 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:40.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T14:29:40.837 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:40.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T14:29:40.908 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:40.908 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T14:29:40.963 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T14:29:40.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T14:29:41.032 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2023-11-28T14:29:41.032 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQA/+WVlC1P9KhAA3P/C2qtV/NZ77fz8lqLkqA== 2023-11-28T14:29:41.032 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2023-11-28T14:29:41.032 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBF+WVlVcHUMxAA0gzl6m9psiIPIP9hYJinCQ== 2023-11-28T14:29:41.032 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2023-11-28T14:29:41.033 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBL+WVlvCEKDBAAcVjmuZiILsSQmVeopwR/VA== 2023-11-28T14:29:41.033 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2023-11-28T14:29:41.033 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBQ+WVlI+ToIRAA+gXfnQAsuTG1BH/MFhNc3A== 2023-11-28T14:29:41.033 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2023-11-28T14:29:41.033 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQA/+WVliBw5LhAAVDVzjm22t+3YU/7iAHHj/w== 2023-11-28T14:29:41.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T14:29:41.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:29:41.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:29:41.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T14:29:41.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T14:29:41.386 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T14:29:41.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T14:29:41.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T14:29:41.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:41.524+0000 7f0ab903a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:41.524+0000 7f0ab903a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T14:29:41.593 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T14:29:41.647 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T14:29:41.647 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T14:29:41.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T14:29:41.690 INFO:tasks.ceph.mon.a:Started 2023-11-28T14:29:41.690 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T14:29:41.690 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T14:29:41.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T14:29:41.692 INFO:tasks.ceph.mgr.x:Started 2023-11-28T14:29:41.692 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T14:29:41.692 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T14:29:41.778 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:41.776+0000 7fb1e401bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.779 INFO:tasks.ceph.mon.a.smithi038.stderr:2023-11-28T14:29:41.778+0000 7fe0fb022980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.779 INFO:tasks.ceph.mon.a.smithi038.stderr:2023-11-28T14:29:41.778+0000 7fe0fb022980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.779 INFO:tasks.ceph.mon.a.smithi038.stderr:2023-11-28T14:29:41.778+0000 7fe0fb022980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:41.801+0000 7f6252ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.836 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:41.835+0000 7f6252ef0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2023-11-28T14:29:41.918 INFO:tasks.ceph:check_enable_crimson: True 2023-11-28T14:29:41.919 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-28T14:29:41.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-28T14:29:41.982 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:41.980+0000 7fb1e401bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:41.982 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:41.980+0000 7fb1e401bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:42.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:42.023+0000 7f5c23f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:42.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:42.029+0000 7f5c23f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:42.111 INFO:tasks.ceph.mon.a.smithi038.stderr:2023-11-28T14:29:42.110+0000 7fe0f2ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-28T14:29:42.111 INFO:tasks.ceph.mon.a.smithi038.stderr:Please be aware that this feature is experimental, untested, 2023-11-28T14:29:42.112 INFO:tasks.ceph.mon.a.smithi038.stderr:unsupported, and may result in data corruption, data loss, 2023-11-28T14:29:42.112 INFO:tasks.ceph.mon.a.smithi038.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-28T14:29:42.112 INFO:tasks.ceph.mon.a.smithi038.stderr:feature with important data. 2023-11-28T14:29:42.112 INFO:tasks.ceph.mon.a.smithi038.stderr: 2023-11-28T14:29:42.457 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:42.455+0000 7fb1e401bf00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T14:29:42.510 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:42.509+0000 7fb1e401bf00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T14:29:42.920 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T14:29:42.920 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:42.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T14:29:42.951 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:42.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T14:29:43.022 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:43.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T14:29:43.060 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.059+0000 7fb1e401bf00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T14:29:43.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd new 12b39dc2-4141-40ea-8259-ba8ada3b2ebe 0 2023-11-28T14:29:43.116 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.115+0000 7fb1e401bf00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T14:29:43.164 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.163+0000 7fb1e401bf00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T14:29:43.222 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.221+0000 7fb1e401bf00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T14:29:43.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:43.220+0000 7f2262224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:43.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:43.227+0000 7f2262224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:43.267 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.266+0000 7fb1e401bf00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T14:29:43.308 INFO:teuthology.orchestra.run.smithi038.stdout:0 2023-11-28T14:29:43.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd new 6de5cc9e-af2e-4c19-8d61-b9f049f8aa54 1 2023-11-28T14:29:43.375 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.374+0000 7fb1e401bf00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T14:29:43.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:43.423+0000 7f0cd091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:43.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:43.430+0000 7f0cd091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:43.452 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.450+0000 7fb1e401bf00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T14:29:43.500 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.499+0000 7fb1e401bf00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T14:29:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:1 2023-11-28T14:29:43.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd new 1adf8329-bb87-4f00-817c-bad7521a3ac5 2 2023-11-28T14:29:43.557 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.555+0000 7fb1e401bf00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T14:29:43.608 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:43.607+0000 7fb1e401bf00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T14:29:43.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:43.627+0000 7ff47042d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:43.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:43.634+0000 7ff47042d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:43.714 INFO:teuthology.orchestra.run.smithi038.stdout:2 2023-11-28T14:29:43.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-28T14:29:43.801 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T14:29:43.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-28T14:29:43.844 INFO:tasks.ceph.osd.0:Started 2023-11-28T14:29:43.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-28T14:29:43.879 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T14:29:43.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-28T14:29:43.922 INFO:tasks.ceph.osd.1:Started 2023-11-28T14:29:43.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-28T14:29:43.960 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T14:29:43.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-28T14:29:44.004 INFO:tasks.ceph.osd.2:Started 2023-11-28T14:29:44.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T14:29:44.033 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.032+0000 7fb1e401bf00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T14:29:44.137 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:44.131+0000 7f3f9715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:44.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:44.139+0000 7f3f9715e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:44.213 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":6,"fsid":"e230aabf-8481-4c61-a863-2290fb6e5a4e","created":"2023-11-28T14:29:41.815686+0000","modified":"2023-11-28T14:29:43.712918+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T14:29:43.712918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"12b39dc2-4141-40ea-8259-ba8ada3b2ebe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6de5cc9e-af2e-4c19-8d61-b9f049f8aa54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1adf8329-bb87-4f00-817c-bad7521a3ac5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:29:44.230 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T14:29:44.231 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T14:29:44.257 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.255+0000 7fb1e401bf00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T14:29:44.410 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.409+0000 7fb1e401bf00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T14:29:44.463 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.461+0000 7fb1e401bf00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T14:29:44.532 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T14:29:44.572 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.571+0000 7fb1e401bf00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T14:29:44.633 INFO:teuthology.misc.health.smithi038.stderr:2023-11-28T14:29:44.628+0000 7f26e8c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:44.635 INFO:teuthology.misc.health.smithi038.stderr:2023-11-28T14:29:44.633+0000 7f26e8c0f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:44.678 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.677+0000 7fb1e401bf00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T14:29:44.703 INFO:teuthology.misc.health.smithi038.stdout: 2023-11-28T14:29:44.703 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":6,"fsid":"e230aabf-8481-4c61-a863-2290fb6e5a4e","created":"2023-11-28T14:29:41.815686+0000","modified":"2023-11-28T14:29:43.712918+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T14:29:43.712918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"12b39dc2-4141-40ea-8259-ba8ada3b2ebe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6de5cc9e-af2e-4c19-8d61-b9f049f8aa54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1adf8329-bb87-4f00-817c-bad7521a3ac5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:29:44.713 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-28T14:29:44.834 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.833+0000 7fb1e401bf00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T14:29:44.881 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:44.880+0000 7fb1e401bf00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T14:29:45.037 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:45.036+0000 7fb1e401bf00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T14:29:45.897 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:45.895+0000 7fb1e401bf00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T14:29:46.209 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:46.208+0000 7fb1e401bf00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T14:29:46.267 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:46.265+0000 7fb1e401bf00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T14:29:46.381 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:46.379+0000 7fb1e401bf00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T14:29:47.401 INFO:tasks.ceph.mgr.x.smithi038.stderr:2023-11-28T14:29:47.400+0000 7fb184182640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-28T14:29:51.015 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T14:29:51.124 INFO:teuthology.misc.health.smithi038.stderr:2023-11-28T14:29:51.120+0000 7f9f680de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:51.126 INFO:teuthology.misc.health.smithi038.stderr:2023-11-28T14:29:51.126+0000 7f9f680de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:51.310 INFO:teuthology.misc.health.smithi038.stdout: 2023-11-28T14:29:51.310 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":9,"fsid":"e230aabf-8481-4c61-a863-2290fb6e5a4e","created":"2023-11-28T14:29:41.815686+0000","modified":"2023-11-28T14:29:50.501903+0000","last_up_change":"2023-11-28T14:29:49.409891+0000","last_in_change":"2023-11-28T14:29:43.712918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T14:29:50.451715+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12b39dc2-4141-40ea-8259-ba8ada3b2ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":75761974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":75761974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":75761974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":75761974}]},"public_addr":"172.21.15.38:6803/75761974","cluster_addr":"172.21.15.38:6802/75761974","heartbeat_back_addr":"172.21.15.38:6808/75761974","heartbeat_front_addr":"172.21.15.38:6809/75761974","state":["exists","up"]},{"osd":1,"uuid":"6de5cc9e-af2e-4c19-8d61-b9f049f8aa54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2650251939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2650251939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2650251939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":2650251939}]},"public_addr":"172.21.15.38:6805/2650251939","cluster_addr":"172.21.15.38:6804/2650251939","heartbeat_back_addr":"172.21.15.38:6810/2650251939","heartbeat_front_addr":"172.21.15.38:6811/2650251939","state":["exists","up"]},{"osd":2,"uuid":"1adf8329-bb87-4f00-817c-bad7521a3ac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":1850121074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1850121074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":1850121074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1850121074}]},"public_addr":"172.21.15.38:6807/1850121074","cluster_addr":"172.21.15.38:6806/1850121074","heartbeat_back_addr":"172.21.15.38:6813/1850121074","heartbeat_front_addr":"172.21.15.38:6812/1850121074","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:29:51.320 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-28T14:29:51.321 INFO:tasks.ceph:Creating RBD pool 2023-11-28T14:29:51.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-28T14:29:51.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:51.425+0000 7f2645cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:51.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:51.433+0000 7f2645cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:52.525 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2023-11-28T14:29:52.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-28T14:29:52.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:52.654+0000 7fe2d895e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:52.664 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:52.663+0000 7fe2d895e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:53.527 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-28T14:29:53.548 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T14:29:53.548 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T14:29:53.549 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T14:29:53.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T14:29:53.549 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T14:29:53.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:53.691+0000 7fc7a5824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:53.696+0000 7fc7a5824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:53.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:53.941 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":812616201},{"type":"v1","addr":"172.21.15.38:6801","nonce":812616201}]},"active_addr":"172.21.15.38:6801/812616201","active_change":"2023-11-28T14:29:46.392968+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":4114426291}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3715287484}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2079260271}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":4262730094}]}]} 2023-11-28T14:29:53.944 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T14:29:53.944 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T14:29:53.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T14:29:54.056 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.052+0000 7f06ed79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.057+0000 7f06ed79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.242 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:54.243 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":12,"fsid":"e230aabf-8481-4c61-a863-2290fb6e5a4e","created":"2023-11-28T14:29:41.815686+0000","modified":"2023-11-28T14:29:53.523999+0000","last_up_change":"2023-11-28T14:29:49.409891+0000","last_in_change":"2023-11-28T14:29:43.712918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T14:29:50.451715+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T14:29:51.756478+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12b39dc2-4141-40ea-8259-ba8ada3b2ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":75761974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":75761974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":75761974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":75761974}]},"public_addr":"172.21.15.38:6803/75761974","cluster_addr":"172.21.15.38:6802/75761974","heartbeat_back_addr":"172.21.15.38:6808/75761974","heartbeat_front_addr":"172.21.15.38:6809/75761974","state":["exists","up"]},{"osd":1,"uuid":"6de5cc9e-af2e-4c19-8d61-b9f049f8aa54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2650251939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2650251939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2650251939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":2650251939}]},"public_addr":"172.21.15.38:6805/2650251939","cluster_addr":"172.21.15.38:6804/2650251939","heartbeat_back_addr":"172.21.15.38:6810/2650251939","heartbeat_front_addr":"172.21.15.38:6811/2650251939","state":["exists","up"]},{"osd":2,"uuid":"1adf8329-bb87-4f00-817c-bad7521a3ac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":1850121074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1850121074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":1850121074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1850121074}]},"public_addr":"172.21.15.38:6807/1850121074","cluster_addr":"172.21.15.38:6806/1850121074","heartbeat_back_addr":"172.21.15.38:6813/1850121074","heartbeat_front_addr":"172.21.15.38:6812/1850121074","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:29:54.254 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T14:29:54.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T14:29:54.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.363+0000 7f7a0c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.368+0000 7f7a0c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.562 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:54.562 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"fsid":"e230aabf-8481-4c61-a863-2290fb6e5a4e","created":"2023-11-28T14:29:41.815686+0000","modified":"2023-11-28T14:29:54.537449+0000","last_up_change":"2023-11-28T14:29:49.409891+0000","last_in_change":"2023-11-28T14:29:43.712918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T14:29:50.451715+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-28T14:29:51.756478+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12b39dc2-4141-40ea-8259-ba8ada3b2ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":75761974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":75761974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":75761974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":75761974}]},"public_addr":"172.21.15.38:6803/75761974","cluster_addr":"172.21.15.38:6802/75761974","heartbeat_back_addr":"172.21.15.38:6808/75761974","heartbeat_front_addr":"172.21.15.38:6809/75761974","state":["exists","up"]},{"osd":1,"uuid":"6de5cc9e-af2e-4c19-8d61-b9f049f8aa54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2650251939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2650251939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2650251939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":2650251939}]},"public_addr":"172.21.15.38:6805/2650251939","cluster_addr":"172.21.15.38:6804/2650251939","heartbeat_back_addr":"172.21.15.38:6810/2650251939","heartbeat_front_addr":"172.21.15.38:6811/2650251939","state":["exists","up"]},{"osd":2,"uuid":"1adf8329-bb87-4f00-817c-bad7521a3ac5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":1850121074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1850121074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":1850121074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1850121074}]},"public_addr":"172.21.15.38:6807/1850121074","cluster_addr":"172.21.15.38:6806/1850121074","heartbeat_back_addr":"172.21.15.38:6813/1850121074","heartbeat_front_addr":"172.21.15.38:6812/1850121074","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T14:29:54.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T14:29:54.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T14:29:54.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T14:29:54.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.690+0000 7fb6e2b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.690+0000 7f6e7619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.690+0000 7fdbdd75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.696+0000 7f6e7619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.696+0000 7fdbdd75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.696+0000 7fb6e2b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.722 INFO:teuthology.orchestra.run.smithi038.stdout:34359738373 2023-11-28T14:29:54.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T14:29:54.723 INFO:teuthology.orchestra.run.smithi038.stdout:34359738373 2023-11-28T14:29:54.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T14:29:54.728 INFO:teuthology.orchestra.run.smithi038.stdout:34359738373 2023-11-28T14:29:54.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T14:29:54.837 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.832+0000 7ff5a7311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.838 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.838+0000 7ff5a7311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.881+0000 7fe7be91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.881+0000 7fecb4f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.888 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.887+0000 7fe7be91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:54.888 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:54.887+0000 7fecb4f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:55.034 INFO:teuthology.orchestra.run.smithi038.stdout:34359738373 2023-11-28T14:29:55.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-11-28T14:29:55.046 DEBUG:teuthology.parallel:result is None 2023-11-28T14:29:55.076 INFO:teuthology.orchestra.run.smithi038.stdout:34359738373 2023-11-28T14:29:55.082 INFO:teuthology.orchestra.run.smithi038.stdout:34359738372 2023-11-28T14:29:55.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-11-28T14:29:55.089 DEBUG:teuthology.parallel:result is None 2023-11-28T14:29:55.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-11-28T14:29:56.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T14:29:56.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:56.219+0000 7f7d9ac64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:56.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:56.225+0000 7f7d9ac64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:56.412 INFO:teuthology.orchestra.run.smithi038.stdout:34359738372 2023-11-28T14:29:56.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-11-28T14:29:57.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T14:29:57.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:57.547+0000 7fc6f9d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:57.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:57.554+0000 7fc6f8aaf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:57.739 INFO:teuthology.orchestra.run.smithi038.stdout:34359738373 2023-11-28T14:29:57.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-11-28T14:29:57.751 DEBUG:teuthology.parallel:result is None 2023-11-28T14:29:57.751 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T14:29:57.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:29:57.864 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:57.860+0000 7f7e4019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:57.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:57.865+0000 7f7e4019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:58.051 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:58.052 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-11-28T14:29:58.063 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-11-28T14:29:56.397355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":282624,"data_stored":1230665,"data_compressed":60102,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.940663"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.549071+0000","last_change":"2023-11-28T14:29:53.666436+0000","last_active":"2023-11-28T14:29:54.549071+0000","last_peered":"2023-11-28T14:29:54.549071+0000","last_clean":"2023-11-28T14:29:54.549071+0000","last_became_active":"2023-11-28T14:29:53.664241+0000","last_became_peered":"2023-11-28T14:29:53.664241+0000","last_unstale":"2023-11-28T14:29:54.549071+0000","last_undegraded":"2023-11-28T14:29:54.549071+0000","last_fullsized":"2023-11-28T14:29:54.549071+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.547113+0000","last_change":"2023-11-28T14:29:53.645519+0000","last_active":"2023-11-28T14:29:54.547113+0000","last_peered":"2023-11-28T14:29:54.547113+0000","last_clean":"2023-11-28T14:29:54.547113+0000","last_became_active":"2023-11-28T14:29:53.643766+0000","last_became_peered":"2023-11-28T14:29:53.643766+0000","last_unstale":"2023-11-28T14:29:54.547113+0000","last_undegraded":"2023-11-28T14:29:54.547113+0000","last_fullsized":"2023-11-28T14:29:54.547113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.546120+0000","last_change":"2023-11-28T14:29:53.664997+0000","last_active":"2023-11-28T14:29:54.546120+0000","last_peered":"2023-11-28T14:29:54.546120+0000","last_clean":"2023-11-28T14:29:54.546120+0000","last_became_active":"2023-11-28T14:29:53.656352+0000","last_became_peered":"2023-11-28T14:29:53.656352+0000","last_unstale":"2023-11-28T14:29:54.546120+0000","last_undegraded":"2023-11-28T14:29:54.546120+0000","last_fullsized":"2023-11-28T14:29:54.546120+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.547406+0000","last_change":"2023-11-28T14:29:53.626626+0000","last_active":"2023-11-28T14:29:54.547406+0000","last_peered":"2023-11-28T14:29:54.547406+0000","last_clean":"2023-11-28T14:29:54.547406+0000","last_became_active":"2023-11-28T14:29:53.624307+0000","last_became_peered":"2023-11-28T14:29:53.624307+0000","last_unstale":"2023-11-28T14:29:54.547406+0000","last_undegraded":"2023-11-28T14:29:54.547406+0000","last_fullsized":"2023-11-28T14:29:54.547406+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-28T14:29:53.625188+0000","last_change":"2023-11-28T14:29:53.625267+0000","last_active":"2023-11-28T14:29:53.625188+0000","last_peered":"2023-11-28T14:29:53.625188+0000","last_clean":"2023-11-28T14:29:52.515295+0000","last_became_active":"2023-11-28T14:29:53.623367+0000","last_became_peered":"2023-11-28T14:29:53.623367+0000","last_unstale":"2023-11-28T14:29:53.625188+0000","last_undegraded":"2023-11-28T14:29:53.625188+0000","last_fullsized":"2023-11-28T14:29:53.625188+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-28T14:29:53.619589+0000","last_change":"2023-11-28T14:29:53.619643+0000","last_active":"2023-11-28T14:29:53.619589+0000","last_peered":"2023-11-28T14:29:53.619589+0000","last_clean":"2023-11-28T14:29:52.515295+0000","last_became_active":"2023-11-28T14:29:53.617670+0000","last_became_peered":"2023-11-28T14:29:53.617670+0000","last_unstale":"2023-11-28T14:29:53.619589+0000","last_undegraded":"2023-11-28T14:29:53.619589+0000","last_fullsized":"2023-11-28T14:29:53.619589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-28T14:29:53.627299+0000","last_change":"2023-11-28T14:29:53.627431+0000","last_active":"2023-11-28T14:29:53.627299+0000","last_peered":"2023-11-28T14:29:53.627299+0000","last_clean":"2023-11-28T14:29:52.515295+0000","last_became_active":"2023-11-28T14:29:53.624693+0000","last_became_peered":"2023-11-28T14:29:53.624693+0000","last_unstale":"2023-11-28T14:29:53.627299+0000","last_undegraded":"2023-11-28T14:29:53.627299+0000","last_fullsized":"2023-11-28T14:29:53.627299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.545594+0000","last_change":"2023-11-28T14:29:53.634211+0000","last_active":"2023-11-28T14:29:54.545594+0000","last_peered":"2023-11-28T14:29:54.545594+0000","last_clean":"2023-11-28T14:29:54.545594+0000","last_became_active":"2023-11-28T14:29:53.632338+0000","last_became_peered":"2023-11-28T14:29:53.632338+0000","last_unstale":"2023-11-28T14:29:54.545594+0000","last_undegraded":"2023-11-28T14:29:54.545594+0000","last_fullsized":"2023-11-28T14:29:54.545594+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.545032+0000","last_change":"2023-11-28T14:29:51.549486+0000","last_active":"2023-11-28T14:29:54.545032+0000","last_peered":"2023-11-28T14:29:54.545032+0000","last_clean":"2023-11-28T14:29:54.545032+0000","last_became_active":"2023-11-28T14:29:51.545391+0000","last_became_peered":"2023-11-28T14:29:51.545391+0000","last_unstale":"2023-11-28T14:29:54.545032+0000","last_undegraded":"2023-11-28T14:29:54.545032+0000","last_fullsized":"2023-11-28T14:29:54.545032+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:50.501903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:50.501903+0000","last_clean_scrub_stamp":"2023-11-28T14:29:50.501903+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":114688,"data_stored":607011,"data_compressed":30051,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":114688,"data_stored":607011,"data_compressed":30051,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:29:58.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T14:29:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:58.171+0000 7f3a4d75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:58.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:58.176+0000 7f3a4d75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:58.358 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:58.358 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-11-28T14:29:58.369 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-11-28T14:29:56.397355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":276,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":282624,"data_stored":1230665,"data_compressed":60102,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.940663"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.549071+0000","last_change":"2023-11-28T14:29:53.666436+0000","last_active":"2023-11-28T14:29:54.549071+0000","last_peered":"2023-11-28T14:29:54.549071+0000","last_clean":"2023-11-28T14:29:54.549071+0000","last_became_active":"2023-11-28T14:29:53.664241+0000","last_became_peered":"2023-11-28T14:29:53.664241+0000","last_unstale":"2023-11-28T14:29:54.549071+0000","last_undegraded":"2023-11-28T14:29:54.549071+0000","last_fullsized":"2023-11-28T14:29:54.549071+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.547113+0000","last_change":"2023-11-28T14:29:53.645519+0000","last_active":"2023-11-28T14:29:54.547113+0000","last_peered":"2023-11-28T14:29:54.547113+0000","last_clean":"2023-11-28T14:29:54.547113+0000","last_became_active":"2023-11-28T14:29:53.643766+0000","last_became_peered":"2023-11-28T14:29:53.643766+0000","last_unstale":"2023-11-28T14:29:54.547113+0000","last_undegraded":"2023-11-28T14:29:54.547113+0000","last_fullsized":"2023-11-28T14:29:54.547113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.546120+0000","last_change":"2023-11-28T14:29:53.664997+0000","last_active":"2023-11-28T14:29:54.546120+0000","last_peered":"2023-11-28T14:29:54.546120+0000","last_clean":"2023-11-28T14:29:54.546120+0000","last_became_active":"2023-11-28T14:29:53.656352+0000","last_became_peered":"2023-11-28T14:29:53.656352+0000","last_unstale":"2023-11-28T14:29:54.546120+0000","last_undegraded":"2023-11-28T14:29:54.546120+0000","last_fullsized":"2023-11-28T14:29:54.546120+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.547406+0000","last_change":"2023-11-28T14:29:53.626626+0000","last_active":"2023-11-28T14:29:54.547406+0000","last_peered":"2023-11-28T14:29:54.547406+0000","last_clean":"2023-11-28T14:29:54.547406+0000","last_became_active":"2023-11-28T14:29:53.624307+0000","last_became_peered":"2023-11-28T14:29:53.624307+0000","last_unstale":"2023-11-28T14:29:54.547406+0000","last_undegraded":"2023-11-28T14:29:54.547406+0000","last_fullsized":"2023-11-28T14:29:54.547406+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-28T14:29:53.625188+0000","last_change":"2023-11-28T14:29:53.625267+0000","last_active":"2023-11-28T14:29:53.625188+0000","last_peered":"2023-11-28T14:29:53.625188+0000","last_clean":"2023-11-28T14:29:52.515295+0000","last_became_active":"2023-11-28T14:29:53.623367+0000","last_became_peered":"2023-11-28T14:29:53.623367+0000","last_unstale":"2023-11-28T14:29:53.625188+0000","last_undegraded":"2023-11-28T14:29:53.625188+0000","last_fullsized":"2023-11-28T14:29:53.625188+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-28T14:29:53.619589+0000","last_change":"2023-11-28T14:29:53.619643+0000","last_active":"2023-11-28T14:29:53.619589+0000","last_peered":"2023-11-28T14:29:53.619589+0000","last_clean":"2023-11-28T14:29:52.515295+0000","last_became_active":"2023-11-28T14:29:53.617670+0000","last_became_peered":"2023-11-28T14:29:53.617670+0000","last_unstale":"2023-11-28T14:29:53.619589+0000","last_undegraded":"2023-11-28T14:29:53.619589+0000","last_fullsized":"2023-11-28T14:29:53.619589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-28T14:29:53.627299+0000","last_change":"2023-11-28T14:29:53.627431+0000","last_active":"2023-11-28T14:29:53.627299+0000","last_peered":"2023-11-28T14:29:53.627299+0000","last_clean":"2023-11-28T14:29:52.515295+0000","last_became_active":"2023-11-28T14:29:53.624693+0000","last_became_peered":"2023-11-28T14:29:53.624693+0000","last_unstale":"2023-11-28T14:29:53.627299+0000","last_undegraded":"2023-11-28T14:29:53.627299+0000","last_fullsized":"2023-11-28T14:29:53.627299+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.545594+0000","last_change":"2023-11-28T14:29:53.634211+0000","last_active":"2023-11-28T14:29:54.545594+0000","last_peered":"2023-11-28T14:29:54.545594+0000","last_clean":"2023-11-28T14:29:54.545594+0000","last_became_active":"2023-11-28T14:29:53.632338+0000","last_became_peered":"2023-11-28T14:29:53.632338+0000","last_unstale":"2023-11-28T14:29:54.545594+0000","last_undegraded":"2023-11-28T14:29:54.545594+0000","last_fullsized":"2023-11-28T14:29:54.545594+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:52.515295+0000","last_clean_scrub_stamp":"2023-11-28T14:29:52.515295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-28T14:29:54.545032+0000","last_change":"2023-11-28T14:29:51.549486+0000","last_active":"2023-11-28T14:29:54.545032+0000","last_peered":"2023-11-28T14:29:54.545032+0000","last_clean":"2023-11-28T14:29:54.545032+0000","last_became_active":"2023-11-28T14:29:51.545391+0000","last_became_peered":"2023-11-28T14:29:51.545391+0000","last_unstale":"2023-11-28T14:29:54.545032+0000","last_undegraded":"2023-11-28T14:29:54.545032+0000","last_fullsized":"2023-11-28T14:29:54.545032+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T14:29:50.501903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T14:29:50.501903+0000","last_clean_scrub_stamp":"2023-11-28T14:29:50.501903+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":114688,"data_stored":607011,"data_compressed":30051,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":114688,"data_stored":607011,"data_compressed":30051,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-28T14:29:58.370 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T14:29:58.370 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T14:29:58.370 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T14:29:58.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T14:29:58.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:58.477+0000 7fe836a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:58.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:29:58.481+0000 7fe836a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:29:58.700 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:29:58.700 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T14:29:58.712 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T14:29:58.712 INFO:teuthology.run_tasks:Running task cbt... 2023-11-28T14:29:58.726 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi038.front.sepia.ceph.com', 'osds': ['ubuntu@smithi038.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi038.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi038.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi038.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-11-28T14:29:58.729 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-11-28T14:29:58.747 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:29:58.747 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-28T14:29:58.799 DEBUG:teuthology.orchestra.run.smithi038:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-11-28T14:29:58.855 INFO:teuthology.orchestra.run.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-11-28T14:29:59.272 INFO:tasks.cbt:Installing collectl 2023-11-28T14:29:59.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-11-28T14:29:59.379 INFO:teuthology.orchestra.run.smithi038.stderr:--2023-11-28 14:29:59-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-11-28T14:29:59.520 INFO:teuthology.orchestra.run.smithi038.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2023-11-28T14:29:59.536 INFO:teuthology.orchestra.run.smithi038.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-11-28T14:29:59.869 INFO:teuthology.orchestra.run.smithi038.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-28T14:29:59.869 INFO:teuthology.orchestra.run.smithi038.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlZflnAOMMFVAVb-pW0fJw4aSXKNwSTAkYfri49qUusu5iYxYu0IN0F98I-reZSUrDOQ2DvfMPtmAd9rWMz-Vl7ghSNQ%3D%3D&use_mirror=cytranet&r= [following] 2023-11-28T14:29:59.869 INFO:teuthology.orchestra.run.smithi038.stderr:--2023-11-28 14:29:59-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlZflnAOMMFVAVb-pW0fJw4aSXKNwSTAkYfri49qUusu5iYxYu0IN0F98I-reZSUrDOQ2DvfMPtmAd9rWMz-Vl7ghSNQ%3D%3D&use_mirror=cytranet&r= 2023-11-28T14:29:59.870 INFO:teuthology.orchestra.run.smithi038.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-11-28T14:29:59.950 INFO:teuthology.orchestra.run.smithi038.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-11-28T14:30:00.223 INFO:teuthology.orchestra.run.smithi038.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-28T14:30:00.224 INFO:teuthology.orchestra.run.smithi038.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-11-28T14:30:00.224 INFO:teuthology.orchestra.run.smithi038.stderr:--2023-11-28 14:30:00-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-11-28T14:30:00.235 INFO:teuthology.orchestra.run.smithi038.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2023-11-28T14:30:00.306 INFO:teuthology.orchestra.run.smithi038.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2023-11-28T14:30:00.631 INFO:teuthology.orchestra.run.smithi038.stderr:HTTP request sent, awaiting response... 200 OK 2023-11-28T14:30:00.632 INFO:teuthology.orchestra.run.smithi038.stderr:Length: 620693 (606K) [application/x-gzip] 2023-11-28T14:30:00.632 INFO:teuthology.orchestra.run.smithi038.stderr:Saving to: ‘collectl.tar.gz’ 2023-11-28T14:30:00.632 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:30:00.702 INFO:teuthology.orchestra.run.smithi038.stderr: 0K .......... .......... .......... .......... .......... 8% 707K 1s 2023-11-28T14:30:00.771 INFO:teuthology.orchestra.run.smithi038.stderr: 50K .......... .......... .......... .......... .......... 16% 717K 1s 2023-11-28T14:30:00.773 INFO:teuthology.orchestra.run.smithi038.stderr: 100K .......... .......... .......... .......... .......... 24% 40.9M 0s 2023-11-28T14:30:00.842 INFO:teuthology.orchestra.run.smithi038.stderr: 150K .......... .......... .......... .......... .......... 32% 728K 0s 2023-11-28T14:30:00.844 INFO:teuthology.orchestra.run.smithi038.stderr: 200K .......... .......... .......... .......... .......... 41% 44.5M 0s 2023-11-28T14:30:00.844 INFO:teuthology.orchestra.run.smithi038.stderr: 250K .......... .......... .......... .......... .......... 49% 55.9M 0s 2023-11-28T14:30:00.844 INFO:teuthology.orchestra.run.smithi038.stderr: 300K .......... .......... .......... .......... .......... 57% 67.6M 0s 2023-11-28T14:30:00.914 INFO:teuthology.orchestra.run.smithi038.stderr: 350K .......... .......... .......... .......... .......... 65% 743K 0s 2023-11-28T14:30:00.914 INFO:teuthology.orchestra.run.smithi038.stderr: 400K .......... .......... .......... .......... .......... 74% 69.5M 0s 2023-11-28T14:30:00.914 INFO:teuthology.orchestra.run.smithi038.stderr: 450K .......... .......... .......... .......... .......... 82% 77.3M 0s 2023-11-28T14:30:00.914 INFO:teuthology.orchestra.run.smithi038.stderr: 500K .......... .......... .......... .......... .......... 90% 91.2M 0s 2023-11-28T14:30:00.915 INFO:teuthology.orchestra.run.smithi038.stderr: 550K .......... .......... .......... .......... .......... 98% 119M 0s 2023-11-28T14:30:00.915 INFO:teuthology.orchestra.run.smithi038.stderr: 600K ...... 100% 11.4T=0.3s 2023-11-28T14:30:00.915 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:30:00.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28 14:30:00 (2.09 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-11-28T14:30:00.915 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:30:00.959 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/ 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/collectl.conf 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/ 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/SlowProc.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/WhySummary.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Socket.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/colmux.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Tutorial.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Gexpr.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Import.html 2023-11-28T14:30:00.960 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/style.css 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/FileNaming.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/TheMath.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Environmental.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Process.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/HiResTime.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/slub.jpg 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Lustre.html 2023-11-28T14:30:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Documentation.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Inodes.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Graphite.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/InputFiles.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Export.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Interrupts.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Startup.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Performance.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Messages.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Logging.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Data-detail.html 2023-11-28T14:30:00.962 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/CPUs.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Matrix.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Data-brief.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Misc.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Examples.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/OpenStack.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Data.html 2023-11-28T14:30:00.963 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Exceptions.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Playback.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Hello.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Sockets.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Network.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Infiniband.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Memory.html 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-11-28T14:30:00.964 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/Disks.html 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/service/ 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/service/collectl.service 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/initd/ 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/initd/collectl-suse 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/initd/collectl-debian 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/initd/collectl-generic 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/initd/collectl 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/misc.ph 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/gexpr.ph 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/statsd.ph 2023-11-28T14:30:00.965 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/envrules.std 2023-11-28T14:30:00.966 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/hello.ph 2023-11-28T14:30:00.966 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/vmsum.ph 2023-11-28T14:30:00.966 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/proctree.ph 2023-11-28T14:30:00.966 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/README-WINDOWS 2023-11-28T14:30:00.966 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/formatit.ph 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/colmux 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/INSTALL 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/graphite.ph 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/RELEASE-collectl 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/vmstat.ph 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/README 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/man1/ 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/man1/collectl.1 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/man1/colmux.1 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/lexpr.ph 2023-11-28T14:30:00.968 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/client.pl 2023-11-28T14:30:00.969 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/collectl 2023-11-28T14:30:00.969 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/COPYING 2023-11-28T14:30:00.969 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/vnet.ph 2023-11-28T14:30:00.969 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/GPL 2023-11-28T14:30:00.969 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/UNINSTALL 2023-11-28T14:30:00.969 INFO:teuthology.orchestra.run.smithi038.stdout:collectl-4.3.1/ARTISTIC 2023-11-28T14:30:01.025 INFO:teuthology.orchestra.run.smithi038.stderr:./INSTALL: line 90: chkconfig: command not found 2023-11-28T14:30:01.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-11-28T14:30:01.628 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:03:43 ago on Tue 28 Nov 2023 02:26:18 PM UTC. 2023-11-28T14:30:01.732 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-11-28T14:30:01.735 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-11-28T14:30:01.793 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout: librbd-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 23 k 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout: perf x86_64 5.14.0-386.el9 appstream 7.4 M 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-11-28T14:30:01.794 INFO:teuthology.orchestra.run.smithi038.stdout: libradospp-devel x86_64 2:18.0.0-7503.g5537419c.el9 ceph 30 k 2023-11-28T14:30:01.795 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:30:01.795 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-11-28T14:30:01.795 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-11-28T14:30:01.795 INFO:teuthology.orchestra.run.smithi038.stdout:Install 5 Packages 2023-11-28T14:30:01.795 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:30:01.796 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 7.6 M 2023-11-28T14:30:01.796 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 11 M 2023-11-28T14:30:01.796 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-11-28T14:30:02.425 INFO:teuthology.orchestra.run.smithi038.stdout:(1/5): librbd-devel-18.0.0-7503.g5537419c.el9.x 78 kB/s | 23 kB 00:00 2023-11-28T14:30:02.534 INFO:teuthology.orchestra.run.smithi038.stdout:(2/5): libradospp-devel-18.0.0-7503.g5537419c.e 74 kB/s | 30 kB 00:00 2023-11-28T14:30:02.792 INFO:teuthology.orchestra.run.smithi038.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 51 kB/s | 13 kB 00:00 2023-11-28T14:30:02.834 INFO:teuthology.orchestra.run.smithi038.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 290 kB/s | 118 kB 00:00 2023-11-28T14:30:03.085 INFO:teuthology.orchestra.run.smithi038.stdout:(5/5): perf-5.14.0-386.el9.x86_64.rpm 7.8 MB/s | 7.4 MB 00:00 2023-11-28T14:30:03.088 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-11-28T14:30:03.088 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.9 MB/s | 7.6 MB 00:01 2023-11-28T14:30:03.168 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-11-28T14:30:03.191 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-11-28T14:30:03.191 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-11-28T14:30:03.410 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-11-28T14:30:03.411 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-11-28T14:30:03.751 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-11-28T14:30:03.820 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-11-28T14:30:03.851 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-11-28T14:30:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradospp-devel-2:18.0.0-7503.g5537419c.el9.x86_64 3/5 2023-11-28T14:30:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librbd-devel-2:18.0.0-7503.g5537419c.el9.x86_64 4/5 2023-11-28T14:30:04.373 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : perf-5.14.0-386.el9.x86_64 5/5 2023-11-28T14:30:05.425 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: perf-5.14.0-386.el9.x86_64 5/5 2023-11-28T14:30:05.425 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradospp-devel-2:18.0.0-7503.g5537419c.el9.x86_64 1/5 2023-11-28T14:30:05.425 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2/5 2023-11-28T14:30:05.425 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : perf-5.14.0-386.el9.x86_64 3/5 2023-11-28T14:30:05.425 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-11-28T14:30:05.761 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-11-28T14:30:05.761 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:30:05.762 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-11-28T14:30:05.762 INFO:teuthology.orchestra.run.smithi038.stdout: libradospp-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:05.762 INFO:teuthology.orchestra.run.smithi038.stdout: librbd-devel-2:18.0.0-7503.g5537419c.el9.x86_64 2023-11-28T14:30:05.762 INFO:teuthology.orchestra.run.smithi038.stdout: pdsh-2.34-7.el9.x86_64 2023-11-28T14:30:05.762 INFO:teuthology.orchestra.run.smithi038.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-11-28T14:30:05.762 INFO:teuthology.orchestra.run.smithi038.stdout: perf-5.14.0-386.el9.x86_64 2023-11-28T14:30:05.763 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-11-28T14:30:05.763 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-11-28T14:30:06.189 INFO:tasks.cbt:benchmark: radosbench 2023-11-28T14:30:06.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-11-28T14:30:06.222 DEBUG:teuthology.orchestra.run.smithi038:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-28T14:30:06.633 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - Settings.cluster: 2023-11-28T14:30:06.634 INFO:teuthology.orchestra.run.smithi038.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-11-28T14:30:06.634 INFO:teuthology.orchestra.run.smithi038.stderr: 'clients': ['ubuntu@smithi038.front.sepia.ceph.com'], 2023-11-28T14:30:06.634 INFO:teuthology.orchestra.run.smithi038.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-11-28T14:30:06.634 INFO:teuthology.orchestra.run.smithi038.stderr: 'head': 'ubuntu@smithi038.front.sepia.ceph.com', 2023-11-28T14:30:06.634 INFO:teuthology.orchestra.run.smithi038.stderr: 'is_teuthology': True, 2023-11-28T14:30:06.634 INFO:teuthology.orchestra.run.smithi038.stderr: 'iterations': 1, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'mons': ['ubuntu@smithi038.front.sepia.ceph.com'], 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'osds': ['ubuntu@smithi038.front.sepia.ceph.com'], 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'osds_per_node': 3, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'pid_dir': '/var/run/ceph', 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'pgp_size': 256, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'replication': 'replicated'}}, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'rebuild_every_test': False, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'rgws': {'ubuntu@smithi038.front.sepia.ceph.com': None}, 2023-11-28T14:30:06.635 INFO:teuthology.orchestra.run.smithi038.stderr: 'tmp_dir': '/tmp/cbt', 2023-11-28T14:30:06.636 INFO:teuthology.orchestra.run.smithi038.stderr: 'use_existing': True, 2023-11-28T14:30:06.636 INFO:teuthology.orchestra.run.smithi038.stderr: 'user': 'ubuntu'} 2023-11-28T14:30:06.655 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-11-28T14:30:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:30:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:30:06.657 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2023-11-28T14:30:06.660 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:30:06.660 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-28T14:30:06.662 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:30:06.662 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:30:06.662 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2023-11-28T14:30:06.665 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:30:06.665 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-28T14:30:06.665 INFO:teuthology.orchestra.run.smithi038.stderr:14:30:06 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-28T14:31:06.669 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.670 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-28T14:31:06.732 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.733 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-28T14:31:06.785 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.785 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-11-28T14:31:06.787 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.787 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-11-28T14:31:06.788 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.789 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.789 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2023-11-28T14:31:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.811 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/$(basename ${i}).smithi038.front.sepia.ceph.com ; done 2023-11-28T14:31:06.815 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.815 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.815 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-11-28T14:31:06.843 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - WARNING - cbt - sudo killall -9 rados 2023-11-28T14:31:06.843 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - WARNING - cbt - error 1 seen, continuing anyway... 2023-11-28T14:31:06.863 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c. 2023-11-28T14:31:06.863 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-11-28T14:31:06.863 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.863 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.863 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-11-28T14:31:06.882 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.883 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.883 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2023-11-28T14:31:06.885 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.885 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-28T14:31:06.886 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.887 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.887 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2023-11-28T14:31:06.890 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.890 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-28T14:31:06.890 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-28T14:31:06.890 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.890 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:06.890 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-11-28T14:31:07.198 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:07 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:07.198 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:07 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:07.198 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-11-28T14:31:07.998 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:07 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:07 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-11-28T14:31:09.002 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:09 - INFO - cbt - Checking Health after pool creation. 2023-11-28T14:31:14.007 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:14 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:14.007 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:14 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:14.007 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-28T14:31:14.339 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:14 - INFO - cbt - Final Pool Health Check. 2023-11-28T14:31:19.344 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.344 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.345 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-28T14:31:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.665 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-28T14:31:19.694 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.694 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-28T14:31:19.739 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.739 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-11-28T14:31:19.740 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.741 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-11-28T14:31:19.742 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.742 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.742 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-28T14:31:19.886 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:19.886 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-28T14:31:20.492 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:20.492 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:20.493 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-11-28T14:31:20.828 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:20.829 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:20.829 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2023-11-28T14:31:20.870 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:20.871 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:20.871 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2023-11-28T14:31:21.673 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.674 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.674 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2023-11-28T14:31:21.676 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.676 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-28T14:31:21.678 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.678 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.678 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - INFO - cbt - Running radosbench write test. 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.682 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2023-11-28T14:31:21.683 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.683 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:31:21.684 INFO:teuthology.orchestra.run.smithi038.stderr:14:31:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2023-11-28T14:32:22.834 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:22.834 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-28T14:32:22.871 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:22.871 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-28T14:32:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:22.921 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2023-11-28T14:32:22.922 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:22.922 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2023-11-28T14:32:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:22.924 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2023-11-28T14:32:23.238 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.291 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.292 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2023-11-28T14:32:23.312 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.312 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/$(basename ${i}).smithi038.front.sepia.ceph.com ; done 2023-11-28T14:32:23.338 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - INFO - cbt - Convert results to json format. 2023-11-28T14:32:23.338 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.339 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.339 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.339 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-28T14:32:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-28T14:32:23.979 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.979 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:23.979 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-11-28T14:32:24.294 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:24 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:24.294 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:24 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:24.295 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand 2023-11-28T14:32:24.317 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:24 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:24.318 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:24 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:24.318 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/ceph_settings.out 2023-11-28T14:32:25.105 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.105 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.106 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl 2023-11-28T14:32:25.108 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.108 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-28T14:32:25.110 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.110 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.110 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf 2023-11-28T14:32:25.113 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.113 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-28T14:32:25.113 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-28T14:32:25.113 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - INFO - cbt - Running radosbench rand test. 2023-11-28T14:32:25.113 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.114 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.114 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.0 2023-11-28T14:32:25.115 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.115 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:32:25.115 INFO:teuthology.orchestra.run.smithi038.stderr:14:32:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.1 2023-11-28T14:33:26.237 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.237 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-28T14:33:26.285 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.285 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-28T14:33:26.340 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.341 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf.data 2023-11-28T14:33:26.342 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.342 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.* 2023-11-28T14:33:26.344 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.344 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.344 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/historic_ops.out 2023-11-28T14:33:26.659 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.662 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.663 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* 2023-11-28T14:33:26.683 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - get_fqdn_local()=smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.683 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/$(basename ${i}).smithi038.front.sepia.ceph.com ; done 2023-11-28T14:33:26.714 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - INFO - cbt - Convert results to json format. 2023-11-28T14:33:26.714 INFO:teuthology.orchestra.run.smithi038.stderr:14:33:26 - DEBUG - cbt - Nodes : ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:33:26.750 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-11-28T14:33:26.759 DEBUG:teuthology.orchestra.run.smithi038:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-11-28T14:33:26.917 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-11-28T14:33:26.917 INFO:tasks.cbt_performance:reading cbt preferences 2023-11-28T14:33:26.917 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-11-28T14:33:27.062 INFO:teuthology.orchestra.run.smithi038.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi038.front.sepia.ceph.com 2023-11-28T14:33:27.063 INFO:teuthology.orchestra.run.smithi038.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi038.front.sepia.ceph.com 2023-11-28T14:33:27.063 INFO:teuthology.orchestra.run.smithi038.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi038.front.sepia.ceph.com 2023-11-28T14:33:27.063 INFO:teuthology.orchestra.run.smithi038.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi038.front.sepia.ceph.com 2023-11-28T14:33:27.063 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:33:27.064 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi038.front.sepia.ceph.com of=/dev/stdout 2023-11-28T14:33:27.270 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-28T14:33:27.270 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2023-11-28T14:33:27.326 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-28T14:33:27.327 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-28T14:33:27.327 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:33:27.327 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi038.front.sepia.ceph.com of=/dev/stdout 2023-11-28T14:33:27.461 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-28T14:33:27.462 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2023-11-28T14:33:27.518 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-28T14:33:27.518 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-28T14:33:27.518 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:33:27.519 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi038.front.sepia.ceph.com of=/dev/stdout 2023-11-28T14:33:27.574 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-28T14:33:27.574 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2023-11-28T14:33:27.630 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-28T14:33:27.630 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-28T14:33:27.630 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-11-28T14:33:27.631 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi038.front.sepia.ceph.com of=/dev/stdout 2023-11-28T14:33:27.686 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-28T14:33:27.686 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2023-11-28T14:33:27.742 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-28T14:33:27.742 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-28T14:33:27.763 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-28T14:33:27.769 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-28T14:33:27.776 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-28T14:33:27.782 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-28T14:33:27.782 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-28T14:33:27.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-28T14:33:27.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:33:27.897+0000 7f41ea6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:27.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-11-28T14:33:27.905+0000 7f41ea6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-28T14:33:28.107 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-28T14:33:28.107 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-28T14:33:28.108 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-28T14:33:28.108 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T14:33:31.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-28T14:33:34.209 INFO:tasks.ceph.osd.0:Stopped 2023-11-28T14:33:34.209 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-28T14:33:34.209 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T14:33:36.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-28T14:33:36.554 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-11-28T14:33:40.311 INFO:tasks.ceph.osd.1:Stopped 2023-11-28T14:33:40.311 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-28T14:33:40.311 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T14:33:41.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-28T14:33:41.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-11-28T14:33:46.413 INFO:tasks.ceph.osd.2:Stopped 2023-11-28T14:33:46.413 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-28T14:33:46.414 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T14:33:46.414 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T14:33:46.447 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T14:33:46.447 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-28T14:33:46.447 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T14:33:46.447 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T14:33:46.501 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T14:33:46.501 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-28T14:33:46.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-28T14:33:46.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:33:46.602 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-28T14:33:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-28T14:33:46.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:33:46.926 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-28T14:33:47.036 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:33:47.036 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-28T14:33:47.093 INFO:tasks.ceph:Archiving mon data... 2023-11-28T14:33:47.094 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175/data/mon.a.tgz 2023-11-28T14:33:47.094 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2023-11-28T14:33:47.120 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.Sy653wVYIx 2023-11-28T14:33:47.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Sy653wVYIx 2023-11-28T14:33:47.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.Sy653wVYIx 2023-11-28T14:33:47.346 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.Sy653wVYIx is 192KB 2023-11-28T14:33:47.402 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.Sy653wVYIx 2023-11-28T14:33:47.418 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-28T14:33:47.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T14:33:47.488 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-28T14:33:47.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T14:33:47.569 INFO:teuthology.orchestra.run.smithi038.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T14:33:47.573 INFO:tasks.ceph:Archiving crash dumps... 2023-11-28T14:33:47.575 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175/remote/smithi038/crash 2023-11-28T14:33:47.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T14:33:47.643 INFO:tasks.ceph:Compressing logs... 2023-11-28T14:33:47.644 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T14:33:47.736 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84520.log 2023-11-28T14:33:47.736 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-28T14:33:47.736 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.84520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82999.log 2023-11-28T14:33:47.736 INFO:teuthology.orchestra.run.smithi038.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84520.log.gz 2023-11-28T14:33:47.736 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83255.log 2023-11-28T14:33:47.736 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82919.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82999.log: /var/log/ceph/ceph-client.admin.83255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84040.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84102.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82919.log: /var/log/ceph/ceph-client.admin.84040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83297.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.84102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83127.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84154.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.83297.log: /var/log/ceph/ceph-client.admin.83127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83087.log 2023-11-28T14:33:47.737 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84241.log 2023-11-28T14:33:47.743 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.84154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84131.log 2023-11-28T14:33:47.743 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.83087.log: gzip/var/log/ceph/ceph-client.admin.84241.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82267.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.84131.log: /var/log/ceph/ceph-client.admin.81803.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84154.log.gz 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82267.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84131.log.gz 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82676.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.81803.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80432.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82239.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82676.log: /var/log/ceph/ceph.tmp-client.admin.80432.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83381.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84177.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.83381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83126.log 2023-11-28T14:33:47.744 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/ceph-client.admin.84177.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82001.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82835.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.83126.log: /var/log/ceph/ceph-client.admin.82001.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84177.log.gz 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84242.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83339.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82081.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.84242.log: /var/log/ceph/ceph-client.admin.83339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84521.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82793.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.84521.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84521.log.gz 2023-11-28T14:33:47.745 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/ceph-client.admin.82793.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81891.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82041.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82877.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.81891.log: /var/log/ceph/ceph-client.admin.82041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84821.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83001.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.84821.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84821.log.gz 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82732.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81891.log.gz 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.83001.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.82732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83213.log 2023-11-28T14:33:47.746 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-28T14:33:47.747 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.83213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83000.log 2023-11-28T14:33:47.747 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.83000.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82001.log.gz 2023-11-28T14:33:47.747 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82041.log.gz 2023-11-28T14:33:47.747 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82081.log.gz 2023-11-28T14:33:47.815 INFO:teuthology.orchestra.run.smithi038.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.84242.log.gz 2023-11-28T14:33:47.815 INFO:teuthology.orchestra.run.smithi038.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84241.log.gz 2023-11-28T14:33:47.815 INFO:teuthology.orchestra.run.smithi038.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T14:33:47.819 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81803.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82676.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82732.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82793.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82877.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82919.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82835.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83001.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83000.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82999.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83087.log.gz 2023-11-28T14:33:47.824 INFO:teuthology.orchestra.run.smithi038.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83126.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83127.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83213.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.83255.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83297.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83339.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83381.log.gz 2023-11-28T14:33:47.825 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82267.log.gz 2023-11-28T14:33:47.840 INFO:teuthology.orchestra.run.smithi038.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-28T14:33:47.841 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82239.log.gz 2023-11-28T14:33:47.849 INFO:teuthology.orchestra.run.smithi038.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80432.log.gz 2023-11-28T14:33:47.849 INFO:teuthology.orchestra.run.smithi038.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84040.log.gz 2023-11-28T14:33:47.849 INFO:teuthology.orchestra.run.smithi038.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84102.log.gz 2023-11-28T14:33:47.850 INFO:teuthology.orchestra.run.smithi038.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-28T14:33:56.951 INFO:teuthology.orchestra.run.smithi038.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-28T14:33:57.575 INFO:teuthology.orchestra.run.smithi038.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-28T14:33:58.290 INFO:teuthology.orchestra.run.smithi038.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-28T14:33:58.294 INFO:teuthology.orchestra.run.smithi038.stderr: 2023-11-28T14:33:58.294 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m10.593s 2023-11-28T14:33:58.294 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m8.117s 2023-11-28T14:33:58.294 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.386s 2023-11-28T14:33:58.295 INFO:tasks.ceph:Archiving logs... 2023-11-28T14:33:58.295 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175/remote/smithi038/log 2023-11-28T14:33:58.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T14:33:59.436 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T14:33:59.446 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T14:33:59.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T14:33:59.503 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T14:33:59.512 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T14:33:59.513 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T14:33:59.535 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2023-11-28T14:33:59.606 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T14:33:59.606 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2023-11-28T14:33:59.606 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1057us[+1059us] +/- 86ms 2023-11-28T14:33:59.606 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -538us[ -537us] +/- 64ms 2023-11-28T14:33:59.607 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -267us[ -267us] +/- 78ms 2023-11-28T14:33:59.607 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T14:33:59.623 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T14:33:59.633 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T14:33:59.634 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T14:33:59.644 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T14:33:59.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T14:33:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:33:59.894 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T14:33:59.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T14:33:59.916 INFO:teuthology.task.internal:Duration was 838.690518 seconds 2023-11-28T14:33:59.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T14:33:59.930 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T14:33:59.930 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T14:34:00.068 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T14:34:00.488 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T14:34:00.488 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2023-11-28T14:34:00.488 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T14:34:00.510 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T14:34:00.510 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T14:34:00.584 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T14:34:00.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T14:34:00.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T14:34:00.743 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T14:34:00.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T14:34:00.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T14:34:00.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T14:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2023-11-28T14:34:00.896 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T14:34:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T14:34:00.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T14:34:00.928 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T14:34:00.929 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-11-28_14:04:23-crimson-rados-wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing-distro-crimson-smithi/7470175/remote/smithi038 2023-11-28T14:34:00.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T14:34:01.085 INFO:teuthology.task.internal:Removing archive directory... 2023-11-28T14:34:01.085 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T14:34:01.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T14:34:01.156 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T14:34:01.156 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T14:34:01.168 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T14:34:01.169 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T14:34:01.195 INFO:teuthology.orchestra.run.smithi038.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 14:34 /home/ubuntu/cephtest 2023-11-28T14:34:01.195 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T14:34:01.211 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T14:34:01.243 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-testing-11-19-wip-xuxuehan-testing&var-job_id_selected=7470175 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} duration: 838.6905179023743 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-11-28T14:34:01.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T14:34:01.320 INFO:teuthology.run:pass