2024-01-17T15:53:46.080 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-17T15:53:46.080 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868 --name matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi --owner scheduled_matan@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} --verbose -- /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868/orig.config.yaml 2024-01-17T15:53:46.120 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-17T15:53:46.204 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868 branch: wip-matanb-crimson-testing-17.1-crimson-only description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: null first_in_suite: false job_id: '7518868' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-17.1-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: cecb6cb5cdba1d67db289980b0e46cfe0334a30f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: cecb6cb5cdba1d67db289980b0e46cfe0334a30f workunit: branch: wip-matanb-crimson-testing-17.1-crimson-only env: CRIMSON_COMPAT: '1' sha1: cecb6cb5cdba1d67db289980b0e46cfe0334a30f owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1084 sha1: cecb6cb5cdba1d67db289980b0e46cfe0334a30f sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-testing-17.1-crimson-only suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_cecb6cb5cdba1d67db289980b0e46cfe0334a30f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: cecb6cb5cdba1d67db289980b0e46cfe0334a30f targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYIdAtf0hC1h9XNFKhBOKVdEiQJf0jpwrIVgz6Ur+UKI7wahtTYv4ldlxYRYcrzHjS6BeGEqYFKPawh9clE/jQ= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-17_15:47:21 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-17T15:53:46.204 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_cecb6cb5cdba1d67db289980b0e46cfe0334a30f/qa; will attempt to use it 2024-01-17T15:53:46.205 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_cecb6cb5cdba1d67db289980b0e46cfe0334a30f/qa/tasks 2024-01-17T15:53:46.205 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-17T15:53:46.206 INFO:teuthology.task.internal:Checking packages... 2024-01-17T15:53:46.227 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f' 2024-01-17T15:53:46.228 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-17T15:53:46.228 INFO:teuthology.packaging:ref: None 2024-01-17T15:53:46.228 INFO:teuthology.packaging:tag: None 2024-01-17T15:53:46.228 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-17.1-crimson-only 2024-01-17T15:53:46.228 INFO:teuthology.packaging:sha1: cecb6cb5cdba1d67db289980b0e46cfe0334a30f 2024-01-17T15:53:46.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-17.1-crimson-only 2024-01-17T15:53:46.515 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-720.gcecb6cb5 2024-01-17T15:53:46.517 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-17T15:53:46.523 INFO:teuthology.task.internal:no buildpackages task found 2024-01-17T15:53:46.523 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-17T15:53:46.559 INFO:teuthology.task.internal:Saving configuration 2024-01-17T15:53:46.574 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-17T15:53:46.592 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-17T15:53:46.618 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-17 15:48:33.629100', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYIdAtf0hC1h9XNFKhBOKVdEiQJf0jpwrIVgz6Ur+UKI7wahtTYv4ldlxYRYcrzHjS6BeGEqYFKPawh9clE/jQ='} 2024-01-17T15:53:46.618 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-17T15:53:46.624 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-17T15:53:46.624 INFO:teuthology.run_tasks:Running task console_log... 2024-01-17T15:53:46.683 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f682cbb7dc0>, signals=[15]) 2024-01-17T15:53:46.683 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-17T15:53:46.689 INFO:teuthology.task.internal:Opening connections... 2024-01-17T15:53:46.689 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T15:53:46.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-17T15:53:46.792 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-17T15:53:46.798 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-01-17T15:53:46.860 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-01-17T15:53:46.860 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-01-17T15:53:46.916 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-17T15:53:46.917 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-17T15:53:46.918 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-17T15:53:46.918 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-01-17T15:53:46.947 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-17T15:53:46.956 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-17T15:53:46.983 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-17T15:53:46.983 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-01-17T15:53:46.999 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-17T15:53:47.005 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-17T15:53:47.006 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-01-17T15:53:47.065 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-17T15:53:47.097 INFO:teuthology.run_tasks:Running task kernel... 2024-01-17T15:53:47.111 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-17T15:53:47.111 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-17T15:53:47.111 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-17T15:53:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-17T15:53:47.128 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-01-17T15:53:47.185 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-01-17T15:53:47.185 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-01-17T15:53:47.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-01-17T15:53:50.359 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2024-01-17T15:53:51.801 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-01-17T15:53:54.231 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-01-17T15:53:57.846 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 19 MB 00:02 2024-01-17T15:54:01.940 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.9 MB 00:00 2024-01-17T15:54:03.625 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-01-17T15:54:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 20 MB 00:02 2024-01-17T15:54:10.896 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-01-17T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-17T15:54:13.196 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-17T15:54:13.302 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T15:54:13.317 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-408.el9 CentOS-BaseOS 6.0 M 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-408.el9 CentOS-BaseOS 21 M 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-408.el9 CentOS-BaseOS 39 M 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-408.el9 CentOS-BaseOS 33 M 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-01-17T15:54:13.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 99 M 2024-01-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 122 M 2024-01-17T15:54:13.319 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T15:54:13.774 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-408.el9.x86_64.rpm 13 MB/s | 6.0 MB 00:00 2024-01-17T15:54:14.525 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-408.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-01-17T15:54:15.725 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-5.14.0-408.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-01-17T15:54:16.259 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-408.el9.x86_6 13 MB/s | 33 MB 00:02 2024-01-17T15:54:16.262 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T15:54:16.262 INFO:teuthology.orchestra.run.smithi029.stdout:Total 34 MB/s | 99 MB 00:02 2024-01-17T15:54:16.263 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T15:54:16.736 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T15:54:16.737 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T15:54:17.786 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T15:54:17.786 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T15:54:20.312 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T15:54:21.119 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-408.el9.x86_64 1/4 2024-01-17T15:54:21.306 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-17T15:54:24.306 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-17T15:54:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-17T15:54:29.694 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-17T15:54:29.836 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-408.el9.x86_64 4/4 2024-01-17T15:54:34.936 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-17T15:55:07.856 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 4/4 2024-01-17T15:55:07.856 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-17T15:55:07.856 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:55:07.929 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 4/4 2024-01-17T15:55:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-408.el9.x86_64 4/4 2024-01-17T15:55:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-408.el9.x86_64 1/4 2024-01-17T15:55:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-17T15:55:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-408.el9.x86_64 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-408.el9.x86_64 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-408.el9.x86_64 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-408.el9.x86_64 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:55:08.769 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T15:55:09.551 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-01-17T15:55:09.730 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-17T15:55:09.731 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-17T15:55:09.731 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-408.el9.x86_64 2024-01-17T15:55:09.731 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-17T15:55:09.731 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-17T15:55:09.731 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-17T15:55:09.732 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-01-17T15:55:09.732 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-01-17T15:55:09.845 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-17T15:55:09.845 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-17T15:55:09.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-01-17T15:55:09.888 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-01-17T15:55:09.889 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-17T15:55:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T15:55:09.891 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-01-17T15:55:09.893 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-17T15:55:09.893 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-408.el9.x86_64 2024-01-17T15:55:09.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-17T15:55:10.616 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-01-17T15:55:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-17T15:55:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-01-17T15:55:12.230 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-01-17T15:55:12.246 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.5E8YX0DqDO 2024-01-17T15:55:12.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5E8YX0DqDO 2024-01-17T15:55:12.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.5E8YX0DqDO 2024-01-17T15:55:12.467 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.5E8YX0DqDO is 6KB 2024-01-17T15:55:12.518 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.5E8YX0DqDO 2024-01-17T15:55:12.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-01-17T15:55:12.609 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-17T15:55:12.609 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-17T15:55:12.610 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-17T15:55:12.610 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64.conf 2024-01-17T15:55:12.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64 2024-01-17T15:55:12.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-01-17T15:55:42.804 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-17T15:55:42.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-17T15:55:42.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-17T15:56:01.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-01-17T15:56:10.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-17T15:56:10.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-17T15:57:10.280 DEBUG:teuthology.orchestra.remote:timed out 2024-01-17T15:57:22.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-17T15:57:22.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-17T15:57:22.657 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-01-17T15:57:23.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-17T15:57:23.105 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-408.el9.x86_64"... 2024-01-17T15:57:23.105 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-01-17T15:57:23.121 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-408.el9.x86_64 2024-01-17T15:57:23.121 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-408.el9.x86_64 vs 5.14.0-408.el9.x86_64 2024-01-17T15:57:23.121 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-17T15:57:23.121 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-17T15:57:24.122 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-17T15:57:24.123 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-17T15:57:24.212 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-01-17T15:57:24.250 DEBUG:teuthology.parallel:result is None 2024-01-17T15:57:24.251 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-17T15:57:24.258 INFO:teuthology.task.internal:Creating test directory... 2024-01-17T15:57:24.259 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-17T15:57:24.277 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-17T15:57:24.287 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-17T15:57:24.337 INFO:teuthology.task.internal:Creating archive directory... 2024-01-17T15:57:24.337 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-17T15:57:24.373 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-17T15:57:24.379 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-17T15:57:24.379 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-17T15:57:24.452 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-17T15:57:24.467 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-17T15:57:24.468 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-17T15:57:24.475 INFO:teuthology.task.internal:Configuring sudo... 2024-01-17T15:57:24.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-17T15:57:24.506 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-17T15:57:24.515 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-17T15:57:24.515 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-17T15:57:24.562 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-17T15:57:24.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-17T15:57:24.694 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-17T15:57:24.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-17T15:57:24.819 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T15:57:24.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-17T15:57:24.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-01-17T15:57:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-17T15:57:25.345 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-17T15:57:25.352 INFO:teuthology.task.internal:Starting timer... 2024-01-17T15:57:25.352 INFO:teuthology.run_tasks:Running task pcp... 2024-01-17T15:57:25.391 INFO:teuthology.run_tasks:Running task selinux... 2024-01-17T15:57:25.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-01-17T15:57:25.516 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-01-17T15:57:25.517 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-17T15:57:25.517 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-01-17T15:57:25.551 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-01-17T15:57:25.552 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive'} 2024-01-17T15:57:25.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-17T15:57:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-17T15:57:25.621 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-17T15:57:25.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-01-17T15:57:25.703 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-17T15:57:25.713 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-17T15:57:25.733 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-17T15:57:25.735 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-17T16:01:42.276 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com')] 2024-01-17T16:01:42.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-17T16:01:42.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-17T16:01:42.369 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-01-17T16:01:42.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-01-17T16:01:42.447 INFO:teuthology.run_tasks:Running task clock... 2024-01-17T16:01:42.458 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-17T16:01:42.458 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-17T16:01:42.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-17T16:01:42.520 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-17T16:01:42.537 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-17T16:01:42.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-01-17T16:01:42.580 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-01-17T16:01:42.597 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-17T16:01:42.613 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-17T16:01:42.663 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-01-17T16:01:42.666 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-17T16:01:42.666 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-01-17T16:01:42.667 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-17T16:01:42.667 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-17T16:01:42.667 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-17T16:01:42.667 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-17T16:01:42.668 INFO:teuthology.run_tasks:Running task install... 2024-01-17T16:01:42.678 DEBUG:teuthology.task.install:project ceph 2024-01-17T16:01:42.679 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f'}} 2024-01-17T16:01:42.679 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f'} 2024-01-17T16:01:42.679 INFO:teuthology.task.install:Using flavor: crimson 2024-01-17T16:01:42.685 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-17T16:01:42.685 INFO:teuthology.task.install:extra packages: [] 2024-01-17T16:01:42.686 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f', 'tag': None, 'wait_for_package': False} 2024-01-17T16:01:42.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=cecb6cb5cdba1d67db289980b0e46cfe0334a30f 2024-01-17T16:01:42.832 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-17.1-crimson-only/cecb6cb5cdba1d67db289980b0e46cfe0334a30f/centos/9/flavors/crimson/ 2024-01-17T16:01:42.833 INFO:teuthology.task.install.rpm:Package version is 19.0.0-720.gcecb6cb5 2024-01-17T16:01:43.832 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-17.1-crimson-only/cecb6cb5cdba1d67db289980b0e46cfe0334a30f/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-17.1-crimson-only/cecb6cb5cdba1d67db289980b0e46cfe0334a30f/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-17.1-crimson-only/cecb6cb5cdba1d67db289980b0e46cfe0334a30f/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-17T16:01:43.832 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:01:43.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-17T16:01:43.864 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-17T16:01:43.865 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cecb6cb5cdba1d67db289980b0e46cfe0334a30f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-17T16:01:43.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-17T16:01:44.022 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-17T16:01:44.091 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-01-17T16:01:44.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-01-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi029.stdout:73 files removed 2024-01-17T16:01:44.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-01-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 128 kB/s | 83 kB 00:00 2024-01-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 23 kB/s | 9.5 kB 00:00 2024-01-17T16:01:46.390 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-01-17T16:01:47.210 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.4 MB 00:00 2024-01-17T16:01:48.912 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-01-17T16:02:33.227 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 185 kB/s | 8.0 MB 00:43 2024-01-17T16:02:35.129 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-01-17T16:02:40.765 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.9 MB 00:02 2024-01-17T16:03:53.981 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 211 B/s | 15 kB 01:12 2024-01-17T16:03:56.708 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 20 MB 00:02 2024-01-17T16:04:01.777 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-01-17T16:04:04.515 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:04:04.538 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:04:04.538 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:04:04.538 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:04:04.538 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:04:04.538 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 8.4 M 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 3.1 M 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 3.1 M 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 5.1 M 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 18 M 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 25 k 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 752 k 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-17T16:04:04.539 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 497 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 4.5 M 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 45 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 130 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 162 k 2024-01-17T16:04:04.540 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 321 k 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 298 k 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 100 k 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout:Install 27 Packages 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-01-17T16:04:04.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:04:04.542 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 53 M 2024-01-17T16:04:04.542 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:04:12.313 INFO:teuthology.orchestra.run.smithi029.stdout:(1/29): ceph-base-19.0.0-720.gcecb6cb5.el9.x86_ 1.8 MB/s | 5.1 MB 00:02 2024-01-17T16:04:12.405 INFO:teuthology.orchestra.run.smithi029.stdout:(2/29): ceph-selinux-19.0.0-720.gcecb6cb5.el9.x 274 kB/s | 25 kB 00:00 2024-01-17T16:04:12.739 INFO:teuthology.orchestra.run.smithi029.stdout:(3/29): libcephfs2-19.0.0-720.gcecb6cb5.el9.x86 2.2 MB/s | 752 kB 00:00 2024-01-17T16:04:13.064 INFO:teuthology.orchestra.run.smithi029.stdout:(4/29): ceph-radosgw-19.0.0-720.gcecb6cb5.el9.x 2.3 MB/s | 8.4 MB 00:03 2024-01-17T16:04:13.098 INFO:teuthology.orchestra.run.smithi029.stdout:(5/29): libradosstriper1-19.0.0-720.gcecb6cb5.e 1.4 MB/s | 497 kB 00:00 2024-01-17T16:04:13.198 INFO:teuthology.orchestra.run.smithi029.stdout:(6/29): python3-ceph-argparse-19.0.0-720.gcecb6 450 kB/s | 45 kB 00:00 2024-01-17T16:04:13.307 INFO:teuthology.orchestra.run.smithi029.stdout:(7/29): python3-ceph-common-19.0.0-720.gcecb6cb 1.2 MB/s | 130 kB 00:00 2024-01-17T16:04:13.416 INFO:teuthology.orchestra.run.smithi029.stdout:(8/29): python3-cephfs-19.0.0-720.gcecb6cb5.el9 1.5 MB/s | 162 kB 00:00 2024-01-17T16:04:13.584 INFO:teuthology.orchestra.run.smithi029.stdout:(9/29): python3-rados-19.0.0-720.gcecb6cb5.el9. 1.9 MB/s | 321 kB 00:00 2024-01-17T16:04:13.761 INFO:teuthology.orchestra.run.smithi029.stdout:(10/29): python3-rbd-19.0.0-720.gcecb6cb5.el9.x 1.6 MB/s | 298 kB 00:00 2024-01-17T16:04:13.861 INFO:teuthology.orchestra.run.smithi029.stdout:(11/29): python3-rgw-19.0.0-720.gcecb6cb5.el9.x 995 kB/s | 100 kB 00:00 2024-01-17T16:04:13.987 INFO:teuthology.orchestra.run.smithi029.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 845 kB/s | 106 kB 00:00 2024-01-17T16:04:14.095 INFO:teuthology.orchestra.run.smithi029.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-01-17T16:04:14.162 INFO:teuthology.orchestra.run.smithi029.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-01-17T16:04:14.447 INFO:teuthology.orchestra.run.smithi029.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 661 kB 00:00 2024-01-17T16:04:14.572 INFO:teuthology.orchestra.run.smithi029.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-01-17T16:04:14.623 INFO:teuthology.orchestra.run.smithi029.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 822 kB/s | 41 kB 00:00 2024-01-17T16:04:15.032 INFO:teuthology.orchestra.run.smithi029.stdout:(18/29): librgw2-19.0.0-720.gcecb6cb5.el9.x86_6 2.3 MB/s | 4.5 MB 00:01 2024-01-17T16:04:15.215 INFO:teuthology.orchestra.run.smithi029.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 521 kB/s | 309 kB 00:00 2024-01-17T16:04:15.316 INFO:teuthology.orchestra.run.smithi029.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-01-17T16:04:15.425 INFO:teuthology.orchestra.run.smithi029.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-01-17T16:04:15.534 INFO:teuthology.orchestra.run.smithi029.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-01-17T16:04:16.251 INFO:teuthology.orchestra.run.smithi029.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-01-17T16:04:16.343 INFO:teuthology.orchestra.run.smithi029.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-01-17T16:04:16.453 INFO:teuthology.orchestra.run.smithi029.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 948 kB/s | 192 kB 00:00 2024-01-17T16:04:16.654 INFO:teuthology.orchestra.run.smithi029.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-01-17T16:04:18.454 INFO:teuthology.orchestra.run.smithi029.stdout:(27/29): librados2-19.0.0-720.gcecb6cb5.el9.x86 1.6 MB/s | 3.1 MB 00:02 2024-01-17T16:04:19.271 INFO:teuthology.orchestra.run.smithi029.stdout:(28/29): librbd1-19.0.0-720.gcecb6cb5.el9.x86_6 1.2 MB/s | 3.1 MB 00:02 2024-01-17T16:04:43.921 INFO:teuthology.orchestra.run.smithi029.stdout:(29/29): ceph-common-19.0.0-720.gcecb6cb5.el9.x 522 kB/s | 18 MB 00:34 2024-01-17T16:04:43.925 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:04:43.925 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.3 MB/s | 53 MB 00:39 2024-01-17T16:04:43.925 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:04:43.962 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:04:43.962 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:04:44.421 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:04:44.421 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:04:45.801 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:04:46.180 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-17T16:04:46.318 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/31 2024-01-17T16:04:46.411 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/31 2024-01-17T16:04:46.504 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-720.gcecb6cb5.el9.x86_64 3/31 2024-01-17T16:04:46.537 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-720.gcecb6cb5.el9.x86_64 4/31 2024-01-17T16:04:46.621 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-720.gcecb6cb5.el9.x86_64 4/31 2024-01-17T16:04:46.658 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-17T16:04:46.692 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-17T16:04:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-17T16:04:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-720.gcecb6cb5.el9.x 8/31 2024-01-17T16:04:46.864 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-720.gcecb6cb5.el9.x86_64 9/31 2024-01-17T16:04:46.894 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-720.gcecb6cb5.el9.x86_64 10/31 2024-01-17T16:04:46.982 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-720.gcecb6cb5.el9.x86_64 10/31 2024-01-17T16:04:47.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-17T16:04:47.138 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-17T16:04:47.171 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-17T16:04:47.220 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-17T16:04:47.262 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-17T16:04:47.300 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-17T16:04:47.335 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-17T16:04:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-17T16:04:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-720.gcecb6cb5.el9.x86_64 19/31 2024-01-17T16:04:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-720.gcecb6cb5.el9.x86_64 19/31 2024-01-17T16:04:47.830 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 20/31 2024-01-17T16:04:47.875 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-17T16:04:48.340 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-17T16:04:48.372 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-720.gcecb6cb5.el9.x86_64 23/31 2024-01-17T16:04:48.444 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-720.gcecb6cb5.el9.x86_64 23/31 2024-01-17T16:04:48.523 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 24/31 2024-01-17T16:04:48.551 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86 25/31 2024-01-17T16:04:50.756 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86_64 26/31 2024-01-17T16:04:50.796 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86_64 26/31 2024-01-17T16:04:51.491 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86_64 26/31 2024-01-17T16:04:51.545 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-720.gcecb6cb5.el9.x86_64 27/31 2024-01-17T16:04:51.649 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-720.gcecb6cb5.el9.x86_64 27/31 2024-01-17T16:04:51.649 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-17T16:04:51.650 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-17T16:04:51.650 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:04:51.690 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-720.gcecb6cb5.el9.x86_64 28/31 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-720.gcecb6cb5.el9.x86_64 28/31 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-01-17T16:05:02.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:05:03.864 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 29/31 2024-01-17T16:05:03.887 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 29/31 2024-01-17T16:05:03.887 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:05:03.887 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-17T16:05:03.888 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-17T16:05:03.888 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-17T16:05:03.888 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:05:03.889 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-17T16:05:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-17T16:05:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-17T16:05:05.426 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-17T16:05:05.426 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/31 2024-01-17T16:05:05.426 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/31 2024-01-17T16:05:05.426 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 3/31 2024-01-17T16:05:05.426 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-720.gcecb6cb5.el9.x86_64 4/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-720.gcecb6cb5.el9.x86_64 5/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-720.gcecb6cb5.el9.x86_64 6/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-720.gcecb6cb5.el9.x86_64 7/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-720.gcecb6cb5.el9.x 8/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86 9/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-720.gcecb6cb5.el9.x86_64 10/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-720.gcecb6cb5.el9.x86_64 11/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 12/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 13/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-17T16:05:05.427 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-720.gcecb6cb5.el9.x86_64 28/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-17T16:05:05.428 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-720.gcecb6cb5.el9.x86_64 30/31 2024-01-17T16:05:06.448 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-17T16:05:06.448 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-17T16:05:06.449 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:05:06.451 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-01-17T16:05:06.451 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-17T16:05:06.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:05:06.451 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:05:06.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-01-17T16:05:07.446 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:01:06 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:05:07.610 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 119 M 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-01-17T16:05:07.611 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 158 k 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout:Install 10 Packages 2024-01-17T16:05:07.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:05:07.613 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 122 M 2024-01-17T16:05:07.613 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.1 G 2024-01-17T16:05:07.613 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:05:08.232 INFO:teuthology.orchestra.run.smithi029.stdout:(1/10): libasan-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 417 kB 00:00 2024-01-17T16:05:08.290 INFO:teuthology.orchestra.run.smithi029.stdout:(2/10): libubsan-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-01-17T16:05:08.357 INFO:teuthology.orchestra.run.smithi029.stdout:(3/10): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-01-17T16:05:08.416 INFO:teuthology.orchestra.run.smithi029.stdout:(4/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-01-17T16:05:08.475 INFO:teuthology.orchestra.run.smithi029.stdout:(5/10): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-01-17T16:05:08.526 INFO:teuthology.orchestra.run.smithi029.stdout:(6/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-01-17T16:05:08.843 INFO:teuthology.orchestra.run.smithi029.stdout:(7/10): cryptopp-8.6.0-1.el9.x86_64.rpm 4.0 MB/s | 1.3 MB 00:00 2024-01-17T16:05:08.877 INFO:teuthology.orchestra.run.smithi029.stdout:(8/10): libcephsqlite-19.0.0-720.gcecb6cb5.el9. 174 kB/s | 158 kB 00:00 2024-01-17T16:05:08.902 INFO:teuthology.orchestra.run.smithi029.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-01-17T16:07:39.477 INFO:teuthology.orchestra.run.smithi029.stdout:(10/10): ceph-test-19.0.0-720.gcecb6cb5.el9.x86 808 kB/s | 119 MB 02:31 2024-01-17T16:07:39.480 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:07:39.480 INFO:teuthology.orchestra.run.smithi029.stdout:Total 825 kB/s | 122 MB 02:31 2024-01-17T16:07:39.480 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:07:39.504 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:07:39.504 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:07:40.373 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:07:40.373 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:07:41.540 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:07:41.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-17T16:07:41.686 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-17T16:07:41.718 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/10 2024-01-17T16:07:41.755 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 4/10 2024-01-17T16:07:41.800 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2024-01-17T16:07:41.844 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/10 2024-01-17T16:07:41.885 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 7/10 2024-01-17T16:07:41.926 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libasan-11.4.1-3.el9.x86_64 8/10 2024-01-17T16:07:41.954 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-720.gcecb6cb5.el9.x86_64 9/10 2024-01-17T16:07:59.429 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-720.gcecb6cb5.el9.x86_64 9/10 2024-01-17T16:07:59.509 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-720.gcecb6cb5.el9.x86_64 10/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:19.0.0-720.gcecb6cb5.el9.x86_64 10/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/10 2024-01-17T16:08:15.867 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 7/10 2024-01-17T16:08:15.868 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/10 2024-01-17T16:08:15.868 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-17T16:08:16.326 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-17T16:08:16.326 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:08:16.326 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:08:16.326 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-720.gcecb6cb5.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-17T16:08:16.326 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-17T16:08:16.327 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-720.gcecb6cb5.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-17T16:08:16.327 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-17T16:08:16.327 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-17T16:08:16.327 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:08:16.327 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:08:16.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-01-17T16:08:17.275 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:04:16 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:08:17.445 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:08:17.447 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 6.5 k 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 2.0 M 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 1.5 M 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 245 k 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 4.9 M 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 117 M 2024-01-17T16:08:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-17T16:08:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-17T16:08:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout:Install 36 Packages 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:08:17.451 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 129 M 2024-01-17T16:08:17.452 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.4 G 2024-01-17T16:08:17.452 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:08:17.952 INFO:teuthology.orchestra.run.smithi029.stdout:(1/36): ceph-19.0.0-720.gcecb6cb5.el9.x86_64.rp 16 kB/s | 6.5 kB 00:00 2024-01-17T16:08:19.428 INFO:teuthology.orchestra.run.smithi029.stdout:(2/36): ceph-mds-19.0.0-720.gcecb6cb5.el9.x86_6 1.1 MB/s | 2.0 MB 00:01 2024-01-17T16:08:19.645 INFO:teuthology.orchestra.run.smithi029.stdout:(3/36): ceph-mgr-19.0.0-720.gcecb6cb5.el9.x86_6 722 kB/s | 1.5 MB 00:02 2024-01-17T16:08:19.987 INFO:teuthology.orchestra.run.smithi029.stdout:(4/36): ceph-mgr-modules-core-19.0.0-720.gcecb6 715 kB/s | 245 kB 00:00 2024-01-17T16:08:20.229 INFO:teuthology.orchestra.run.smithi029.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 772 kB/s | 187 kB 00:00 2024-01-17T16:08:20.305 INFO:teuthology.orchestra.run.smithi029.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-01-17T16:08:20.364 INFO:teuthology.orchestra.run.smithi029.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 539 kB/s | 32 kB 00:00 2024-01-17T16:08:20.431 INFO:teuthology.orchestra.run.smithi029.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 711 kB/s | 47 kB 00:00 2024-01-17T16:08:20.498 INFO:teuthology.orchestra.run.smithi029.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 578 kB/s | 39 kB 00:00 2024-01-17T16:08:20.590 INFO:teuthology.orchestra.run.smithi029.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-01-17T16:08:20.758 INFO:teuthology.orchestra.run.smithi029.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 6.9 MB/s | 1.1 MB 00:00 2024-01-17T16:08:20.816 INFO:teuthology.orchestra.run.smithi029.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-01-17T16:08:20.884 INFO:teuthology.orchestra.run.smithi029.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-01-17T16:08:20.951 INFO:teuthology.orchestra.run.smithi029.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 1.6 MB/s | 113 kB 00:00 2024-01-17T16:08:21.020 INFO:teuthology.orchestra.run.smithi029.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-01-17T16:08:21.129 INFO:teuthology.orchestra.run.smithi029.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-01-17T16:08:21.679 INFO:teuthology.orchestra.run.smithi029.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 275 kB/s | 151 kB 00:00 2024-01-17T16:08:21.813 INFO:teuthology.orchestra.run.smithi029.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2024-01-17T16:08:22.056 INFO:teuthology.orchestra.run.smithi029.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 709 kB/s | 172 kB 00:00 2024-01-17T16:08:22.298 INFO:teuthology.orchestra.run.smithi029.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-01-17T16:08:22.515 INFO:teuthology.orchestra.run.smithi029.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 49 kB/s | 11 kB 00:00 2024-01-17T16:08:22.641 INFO:teuthology.orchestra.run.smithi029.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-01-17T16:08:22.758 INFO:teuthology.orchestra.run.smithi029.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-01-17T16:08:22.875 INFO:teuthology.orchestra.run.smithi029.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-01-17T16:08:23.009 INFO:teuthology.orchestra.run.smithi029.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-01-17T16:08:23.139 INFO:teuthology.orchestra.run.smithi029.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 356 kB/s | 46 kB 00:00 2024-01-17T16:08:23.273 INFO:teuthology.orchestra.run.smithi029.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 590 kB/s | 79 kB 00:00 2024-01-17T16:08:23.507 INFO:teuthology.orchestra.run.smithi029.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-01-17T16:08:23.617 INFO:teuthology.orchestra.run.smithi029.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2024-01-17T16:08:23.784 INFO:teuthology.orchestra.run.smithi029.stdout:(30/36): ceph-mon-19.0.0-720.gcecb6cb5.el9.x86_ 852 kB/s | 4.9 MB 00:05 2024-01-17T16:08:23.819 INFO:teuthology.orchestra.run.smithi029.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 444 kB/s | 90 kB 00:00 2024-01-17T16:08:24.020 INFO:teuthology.orchestra.run.smithi029.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-01-17T16:08:24.087 INFO:teuthology.orchestra.run.smithi029.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 118 kB/s | 36 kB 00:00 2024-01-17T16:08:24.205 INFO:teuthology.orchestra.run.smithi029.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-01-17T16:08:24.238 INFO:teuthology.orchestra.run.smithi029.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-01-17T16:09:26.293 INFO:teuthology.orchestra.run.smithi029.stdout:(36/36): ceph-osd-19.0.0-720.gcecb6cb5.el9.x86_ 1.7 MB/s | 117 MB 01:06 2024-01-17T16:09:26.296 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:09:26.296 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.9 MB/s | 129 MB 01:08 2024-01-17T16:09:26.312 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:09:26.353 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:09:26.353 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:09:27.432 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:09:27.432 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:09:29.401 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:09:29.471 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-17T16:09:29.506 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-17T16:09:29.534 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-17T16:09:29.617 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-17T16:09:29.660 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-17T16:09:29.693 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-17T16:09:29.790 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-17T16:09:29.876 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-17T16:09:29.925 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-17T16:09:29.999 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-17T16:09:30.061 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-17T16:09:30.107 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-17T16:09:30.175 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-17T16:09:30.247 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-17T16:09:30.447 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-17T16:09:30.505 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-17T16:09:30.544 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-17T16:09:30.587 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-17T16:09:30.626 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-17T16:09:30.707 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-17T16:09:30.740 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-17T16:09:30.774 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-17T16:09:30.815 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-17T16:09:30.859 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-17T16:09:30.893 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-17T16:09:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-17T16:09:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-17T16:09:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-17T16:09:31.272 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-17T16:09:31.415 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-17T16:09:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-720.gcecb6cb5.el9.n 31/36 2024-01-17T16:09:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-720.gcecb6cb5.el9.x86_64 32/36 2024-01-17T16:09:31.646 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-720.gcecb6cb5.el9.x86_64 32/36 2024-01-17T16:09:31.646 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:09:31.646 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-17T16:09:31.646 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-17T16:09:31.646 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-17T16:09:31.646 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:09:51.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-720.gcecb6cb5.el9.x86_64 33/36 2024-01-17T16:09:51.243 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-720.gcecb6cb5.el9.x86_64 33/36 2024-01-17T16:09:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:09:51.243 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-17T16:09:51.244 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-17T16:09:51.244 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-17T16:09:51.244 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:09:51.737 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-720.gcecb6cb5.el9.x86_64 34/36 2024-01-17T16:09:51.758 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-720.gcecb6cb5.el9.x86_64 34/36 2024-01-17T16:09:51.759 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:09:51.759 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-17T16:09:51.759 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-17T16:09:51.759 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-17T16:09:51.759 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:09:51.971 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-720.gcecb6cb5.el9.x86_64 35/36 2024-01-17T16:09:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-720.gcecb6cb5.el9.x86_64 35/36 2024-01-17T16:09:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:09:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-17T16:09:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-17T16:09:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-17T16:09:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:09:52.055 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-720.gcecb6cb5.el9.x86_64 36/36 2024-01-17T16:10:12.401 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:19.0.0-720.gcecb6cb5.el9.x86_64 36/36 2024-01-17T16:10:12.401 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/36 2024-01-17T16:10:12.401 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/36 2024-01-17T16:10:12.401 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-720.gcecb6cb5.el9.x86_64 3/36 2024-01-17T16:10:12.401 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-720.gcecb6cb5.el9.x86_64 4/36 2024-01-17T16:10:12.401 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-720.gcecb6cb5.el9.x86_64 5/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-720.gcecb6cb5.el9.n 6/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-17T16:10:12.402 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-17T16:10:12.403 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-17T16:10:12.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-17T16:10:12.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-17T16:10:12.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-17T16:10:12.404 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-01-17T16:10:13.505 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-17T16:10:13.506 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-17T16:10:13.507 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:13.508 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:13.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-01-17T16:10:14.433 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:13 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:14.543 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:10:14.598 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:14.599 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:10:14.600 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:14.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-01-17T16:10:15.252 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:14 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:15.408 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 742 k 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:15.409 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 742 k 2024-01-17T16:10:15.410 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 748 k 2024-01-17T16:10:15.410 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:15.701 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-19.0.0-720.gcecb6cb5.el9.noarch.rpm 2.5 MB/s | 742 kB 00:00 2024-01-17T16:10:15.702 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:15.702 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.5 MB/s | 742 kB 00:00 2024-01-17T16:10:15.702 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:15.708 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:15.708 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:15.714 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:15.714 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:15.790 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:16.293 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:16.388 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:16.863 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:17.165 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:17.166 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:17.166 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:17.166 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:17.166 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:17.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-01-17T16:10:17.990 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:16 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:18.149 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 147 k 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 147 k 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 481 k 2024-01-17T16:10:18.150 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:18.382 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-19.0.0-720.gcecb6cb 636 kB/s | 147 kB 00:00 2024-01-17T16:10:18.383 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:18.383 INFO:teuthology.orchestra.run.smithi029.stdout:Total 632 kB/s | 147 kB 00:00 2024-01-17T16:10:18.383 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:18.391 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:18.391 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:18.445 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:18.445 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:18.589 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:18.667 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-720.gcecb6cb5.e 1/1 2024-01-17T16:10:18.691 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-720.gcecb6cb5.e 1/1 2024-01-17T16:10:18.691 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:10:18.692 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-17T16:10:18.692 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:19.766 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-720.gcecb6cb5.e 1/1 2024-01-17T16:10:19.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:19.766 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:19.766 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:19.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:19.766 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:20.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-01-17T16:10:20.623 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:19 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:10:20.779 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:20.781 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:10:20.781 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:20.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-01-17T16:10:21.431 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:20 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:21.587 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:21.588 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:21.588 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:21.588 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:21.588 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:21.588 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 1.7 M 2024-01-17T16:10:21.588 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 28 k 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 15 k 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout:Install 5 Packages 2024-01-17T16:10:21.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:21.590 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-01-17T16:10:21.590 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 8.4 M 2024-01-17T16:10:21.590 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:22.018 INFO:teuthology.orchestra.run.smithi029.stdout:(1/5): ceph-prometheus-alerts-19.0.0-720.gcecb6 58 kB/s | 15 kB 00:00 2024-01-17T16:10:22.044 INFO:teuthology.orchestra.run.smithi029.stdout:(2/5): ceph-grafana-dashboards-19.0.0-720.gcecb 98 kB/s | 28 kB 00:00 2024-01-17T16:10:22.136 INFO:teuthology.orchestra.run.smithi029.stdout:(3/5): ceph-mgr-dashboard-19.0.0-720.gcecb6cb5. 4.6 MB/s | 1.7 MB 00:00 2024-01-17T16:10:22.203 INFO:teuthology.orchestra.run.smithi029.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 166 kB/s | 31 kB 00:00 2024-01-17T16:10:22.320 INFO:teuthology.orchestra.run.smithi029.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 683 kB/s | 188 kB 00:00 2024-01-17T16:10:22.323 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:22.323 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-01-17T16:10:22.323 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:22.330 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:22.330 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:22.396 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:22.396 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:22.641 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:22.737 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-17T16:10:22.778 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-17T16:10:22.817 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-720.gcecb6cb5.el9.no 3/5 2024-01-17T16:10:23.066 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-720.gcecb6cb5.el9.n 4/5 2024-01-17T16:10:23.111 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-720.gcecb6cb5.el9.noarch 5/5 2024-01-17T16:10:23.769 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-720.gcecb6cb5.el9.noarch 5/5 2024-01-17T16:10:23.769 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-720.gcecb6cb5.el9.n 1/5 2024-01-17T16:10:23.769 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-720.gcecb6cb5.el9.noarch 2/5 2024-01-17T16:10:23.769 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-720.gcecb6cb5.el9.no 3/5 2024-01-17T16:10:23.770 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-17T16:10:24.073 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-17T16:10:24.073 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:24.073 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:24.074 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:24.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-17T16:10:24.935 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:23 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:25.095 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================== 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================== 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 7.4 M 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:10:25.097 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout:=========================================================================================== 2024-01-17T16:10:25.098 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-01-17T16:10:25.099 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:25.099 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2024-01-17T16:10:25.099 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-01-17T16:10:25.099 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:25.311 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 187 kB/s | 32 kB 00:00 2024-01-17T16:10:25.370 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 299 kB/s | 17 kB 00:00 2024-01-17T16:10:25.428 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 590 kB/s | 34 kB 00:00 2024-01-17T16:10:25.529 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.7 MB/s | 3.0 MB 00:00 2024-01-17T16:10:26.063 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.5 MB/s | 4.7 MB 00:00 2024-01-17T16:10:26.163 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 4.8 MB 00:00 2024-01-17T16:10:26.197 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 217 kB 00:00 2024-01-17T16:10:26.473 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 802 kB 00:00 2024-01-17T16:10:26.907 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-01-17T16:10:26.941 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 417 kB/s | 194 kB 00:00 2024-01-17T16:10:27.399 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 499 kB/s | 245 kB 00:00 2024-01-17T16:10:27.685 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-7 2.9 MB/s | 7.4 MB 00:02 2024-01-17T16:10:27.688 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:27.688 INFO:teuthology.orchestra.run.smithi029.stdout:Total 14 MB/s | 37 MB 00:02 2024-01-17T16:10:27.703 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:27.730 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:27.730 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:28.125 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:28.125 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:28.846 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:29.005 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-17T16:10:29.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-17T16:10:29.147 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-17T16:10:29.195 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-17T16:10:29.381 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-17T16:10:29.419 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-17T16:10:29.517 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-17T16:10:30.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-17T16:10:30.344 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-17T16:10:31.667 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-17T16:10:32.765 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-17T16:10:32.802 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-720.gcecb6c 12/12 2024-01-17T16:10:37.024 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-720.gcecb6c 12/12 2024-01-17T16:10:37.024 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-720.gcecb6c 1/12 2024-01-17T16:10:37.024 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-17T16:10:37.024 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-17T16:10:37.024 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-17T16:10:37.024 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-17T16:10:37.025 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-17T16:10:37.025 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-17T16:10:37.025 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-17T16:10:37.025 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-17T16:10:37.025 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-17T16:10:37.025 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-17T16:10:37.500 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-17T16:10:37.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-17T16:10:37.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-17T16:10:37.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-17T16:10:37.501 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-17T16:10:37.501 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:37.501 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:37.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-01-17T16:10:38.467 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:37 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:38.626 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 50 k 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-17T16:10:38.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-01-17T16:10:38.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-01-17T16:10:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-01-17T16:10:38.630 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:38.958 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 168 kB/s | 24 kB 00:00 2024-01-17T16:10:38.983 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2024-01-17T16:10:39.042 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-01-17T16:10:39.067 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-01-17T16:10:39.094 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.9 MB/s | 211 kB 00:00 2024-01-17T16:10:39.120 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 823 kB/s | 43 kB 00:00 2024-01-17T16:10:39.146 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): ceph-mgr-rook-19.0.0-720.gcecb6cb5.el9. 152 kB/s | 50 kB 00:00 2024-01-17T16:10:39.379 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-01-17T16:10:39.405 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-01-17T16:10:39.589 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-01-17T16:10:39.631 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 416 kB/s | 202 kB 00:00 2024-01-17T16:10:39.773 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 487 kB/s | 90 kB 00:00 2024-01-17T16:10:39.965 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-01-17T16:10:39.967 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:39.968 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-01-17T16:10:39.968 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:39.983 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:39.983 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:40.089 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:40.089 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:40.803 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-17T16:10:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-17T16:10:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-17T16:10:40.973 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-17T16:10:41.015 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-17T16:10:41.113 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-17T16:10:41.274 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-17T16:10:41.337 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-17T16:10:42.111 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-17T16:10:42.170 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-17T16:10:42.206 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-17T16:10:42.255 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-17T16:10:42.285 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-720.gcecb6cb5.el9.noarch 13/13 2024-01-17T16:10:43.399 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-720.gcecb6cb5.el9.noarch 13/13 2024-01-17T16:10:43.399 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-720.gcecb6cb5.el9.noarch 1/13 2024-01-17T16:10:43.399 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-17T16:10:43.399 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-17T16:10:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-17T16:10:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-17T16:10:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-17T16:10:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-17T16:10:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-17T16:10:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:43.787 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:44.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-01-17T16:10:44.700 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:43 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:44.874 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 133 k 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-17T16:10:44.875 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================= 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-01-17T16:10:44.876 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:44.877 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2024-01-17T16:10:44.877 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-01-17T16:10:44.877 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:45.557 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 907 kB/s | 228 kB 00:00 2024-01-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.0 MB/s | 5.8 MB 00:00 2024-01-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): ceph-mgr-cephadm-19.0.0-720.gcecb6cb5.el 178 kB/s | 133 kB 00:00 2024-01-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-01-17T16:10:46.269 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 238 kB/s | 58 kB 00:00 2024-01-17T16:10:46.295 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 210 kB/s | 51 kB 00:00 2024-01-17T16:10:46.299 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:46.299 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.8 MB/s | 6.8 MB 00:01 2024-01-17T16:10:46.299 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:46.309 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:46.309 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:46.402 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:46.403 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:46.701 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:46.917 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-17T16:10:46.974 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-17T16:10:47.533 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-17T16:10:47.632 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-17T16:10:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-17T16:10:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 6/6 2024-01-17T16:10:48.949 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 6/6 2024-01-17T16:10:48.950 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 1/6 2024-01-17T16:10:48.950 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-17T16:10:48.950 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-17T16:10:48.950 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-17T16:10:48.950 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-17T16:10:49.266 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-17T16:10:49.267 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-17T16:10:49.267 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-17T16:10:49.267 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:49.267 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:49.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-01-17T16:10:50.146 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:49 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:50.304 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 858 k 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-01-17T16:10:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 937 k 2024-01-17T16:10:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 3.1 M 2024-01-17T16:10:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:50.567 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 302 kB/s | 79 kB 00:00 2024-01-17T16:10:51.151 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-19.0.0-720.gcecb6cb5.el9.x86_6 1.0 MB/s | 858 kB 00:00 2024-01-17T16:10:51.153 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:51.153 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.1 MB/s | 937 kB 00:00 2024-01-17T16:10:51.153 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:51.176 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:51.176 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:51.260 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:51.260 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:51.448 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:51.600 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-17T16:10:51.681 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/2 2024-01-17T16:10:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/2 2024-01-17T16:10:52.438 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/2 2024-01-17T16:10:52.761 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-17T16:10:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:52.762 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-17T16:10:52.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:52.762 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:53.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2024-01-17T16:10:53.642 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:52 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:53.799 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-720.gcecb6cb5.el9 ceph-noarch 262 k 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:10:53.800 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:53.801 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 262 k 2024-01-17T16:10:53.801 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.2 M 2024-01-17T16:10:53.801 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:54.055 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-volume-19.0.0-720.gcecb6cb5.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-17T16:10:54.056 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:54.056 INFO:teuthology.orchestra.run.smithi029.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-17T16:10:54.056 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:54.065 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:54.065 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:54.101 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:54.102 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:54.319 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:54.394 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:54.409 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:54.409 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:10:54.409 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-17T16:10:54.409 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:55.552 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-720.gcecb6cb5.el9.noarch 1/1 2024-01-17T16:10:55.553 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:55.553 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:55.553 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-720.gcecb6cb5.el9.noarch 2024-01-17T16:10:55.553 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:55.553 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:55.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-01-17T16:10:56.512 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:55 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:56.670 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 127 k 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:56.671 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 127 k 2024-01-17T16:10:56.672 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 457 k 2024-01-17T16:10:56.672 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:10:56.904 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-19.0.0-720.gcecb6cb5.el9.x86_64. 547 kB/s | 127 kB 00:00 2024-01-17T16:10:56.904 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:10:56.905 INFO:teuthology.orchestra.run.smithi029.stdout:Total 543 kB/s | 127 kB 00:00 2024-01-17T16:10:56.905 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:10:56.911 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:10:56.911 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:10:56.967 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:10:56.967 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:10:57.114 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:10:57.715 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:10:58.035 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:10:58.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:58.035 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:10:58.035 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:10:58.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:58.036 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:58.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-01-17T16:10:58.932 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:57 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:59.035 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:10:59.090 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:59.091 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:10:59.091 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:10:59.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-01-17T16:10:59.741 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:06:58 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:10:59.901 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:10:59.901 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 31 k 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:10:59.902 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:10:59.903 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 k 2024-01-17T16:10:59.903 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 140 k 2024-01-17T16:10:59.903 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:11:00.222 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-19.0.0-720.gcecb6cb5.el9.x86_64 98 kB/s | 31 kB 00:00 2024-01-17T16:11:00.223 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:11:00.223 INFO:teuthology.orchestra.run.smithi029.stdout:Total 97 kB/s | 31 kB 00:00 2024-01-17T16:11:00.223 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:11:00.227 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:11:00.227 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:11:00.259 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:11:00.259 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:11:00.353 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:11:00.474 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:00.916 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:01.194 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:01.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:01.194 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:11:01.194 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:11:01.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:01.194 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:01.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-01-17T16:11:01.992 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:00 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:02.096 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:11:02.150 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:02.151 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:11:02.152 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:02.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-01-17T16:11:02.810 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:01 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:02.913 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:11:02.966 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:02.968 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:11:02.968 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:03.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-01-17T16:11:03.621 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:02 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:03.724 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:11:03.780 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:03.781 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:11:03.781 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:03.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-01-17T16:11:04.432 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:03 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:04.535 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:11:04.591 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:04.593 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:11:04.593 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:04.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-01-17T16:11:05.247 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:04 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:05.350 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:11:05.404 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:05.405 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:11:05.405 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:05.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-01-17T16:11:06.060 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:05 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:06.163 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 is already installed. 2024-01-17T16:11:06.220 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:06.221 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-01-17T16:11:06.221 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:06.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-01-17T16:11:06.867 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:05 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:07.024 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:07.024 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 86 k 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 86 k 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 256 k 2024-01-17T16:11:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:11:07.247 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-19.0.0-720.gcecb6cb5.el9.x86_64.rpm 388 kB/s | 86 kB 00:00 2024-01-17T16:11:07.248 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:11:07.248 INFO:teuthology.orchestra.run.smithi029.stdout:Total 385 kB/s | 86 kB 00:00 2024-01-17T16:11:07.248 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:11:07.255 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:11:07.255 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:11:07.311 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:11:07.311 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:11:07.448 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:11:07.556 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:07.992 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:08.318 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:11:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:11:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:08.318 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:08.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-01-17T16:11:09.162 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:08 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:09.321 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:09.322 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 3.0 M 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.0 M 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 14 M 2024-01-17T16:11:09.323 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:11:10.773 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-19.0.0-720.gcecb6cb5.el9.x86_64.rpm 2.1 MB/s | 3.0 MB 00:01 2024-01-17T16:11:10.774 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:11:10.774 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.1 MB/s | 3.0 MB 00:01 2024-01-17T16:11:10.775 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:11:10.786 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:11:10.786 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:11:10.864 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:11:10.864 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:11:11.317 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:11:11.383 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:11.406 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:11.407 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-17T16:11:11.407 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-17T16:11:11.407 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-17T16:11:11.407 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-17T16:11:11.407 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:12.762 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:12.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:12.762 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:11:12.762 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:11:12.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:12.762 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:13.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-01-17T16:11:13.634 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:12 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:13.793 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 169 k 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:13.794 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 169 k 2024-01-17T16:11:13.795 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 537 k 2024-01-17T16:11:13.795 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:11:14.032 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-19.0.0-720.gcecb6cb5.el9.x86_64.rpm 710 kB/s | 169 kB 00:00 2024-01-17T16:11:14.033 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:11:14.034 INFO:teuthology.orchestra.run.smithi029.stdout:Total 706 kB/s | 169 kB 00:00 2024-01-17T16:11:14.034 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:11:14.042 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:11:14.042 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:11:14.099 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:11:14.099 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:11:14.249 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:11:14.360 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:14.879 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:15.190 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/1 2024-01-17T16:11:15.191 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:15.191 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:11:15.191 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:11:15.191 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:15.191 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:15.502 DEBUG:teuthology.parallel:result is None 2024-01-17T16:11:15.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=cecb6cb5cdba1d67db289980b0e46cfe0334a30f 2024-01-17T16:11:15.833 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-17T16:11:15.853 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-720.gcecb6cb5.el9 2024-01-17T16:11:15.853 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-720.gcecb6cb5.el9 2024-01-17T16:11:15.854 INFO:teuthology.task.install:The correct ceph version 19.0.0-720.gcecb6cb5 is installed. 2024-01-17T16:11:15.855 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-17T16:11:15.855 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:15.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-17T16:11:15.920 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-17T16:11:15.920 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:15.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-01-17T16:11:15.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-17T16:11:16.054 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-17T16:11:16.055 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:16.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-17T16:11:16.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-17T16:11:16.212 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-17T16:11:16.212 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:16.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-01-17T16:11:16.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-17T16:11:16.305 INFO:teuthology.run_tasks:Running task ceph... 2024-01-17T16:11:16.378 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-17T16:11:16.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-01-17T16:11:16.404 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-17T16:11:16.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-17T16:11:16.471 INFO:tasks.ceph:Creating extra log directories... 2024-01-17T16:11:16.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-17T16:11:16.544 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-17T16:11:16.544 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-17T16:11:16.544 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868', 'branch': 'wip-matanb-crimson-testing-17.1-crimson-only', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': None, 'first_in_suite': False, 'job_id': '7518868', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-17.1-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f'}}, 'workunit': {'branch': 'wip-matanb-crimson-testing-17.1-crimson-only', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1084, 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-testing-17.1-crimson-only', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cecb6cb5cdba1d67db289980b0e46cfe0334a30f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYIdAtf0hC1h9XNFKhBOKVdEiQJf0jpwrIVgz6Ur+UKI7wahtTYv4ldlxYRYcrzHjS6BeGEqYFKPawh9clE/jQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'cecb6cb5cdba1d67db289980b0e46cfe0334a30f', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-17_15:47:21', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-17T16:11:16.544 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-17T16:11:16.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2024-01-17T16:11:16.657 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:16.657 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-01-17T16:11:16.711 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-17T16:11:16.711 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-01-17T16:11:16.765 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-17T16:11:16.765 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-17T16:11:16.765 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-17T16:11:16.765 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-17T16:11:16.765 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-17T16:11:16.765 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-17 16:11:11.652945947 +0000 2024-01-17T16:11:16.766 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-17 16:01:17.749872296 +0000 2024-01-17T16:11:16.766 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-17 16:01:17.749872296 +0000 2024-01-17T16:11:16.766 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-01-17 16:01:17.749872296 +0000 2024-01-17T16:11:16.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-17T16:11:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-17T16:11:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-17T16:11:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.6211e-05 s, 5.3 MB/s 2024-01-17T16:11:16.831 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-17T16:11:16.886 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-01-17T16:11:16.940 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-17T16:11:16.940 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-17T16:11:16.940 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-01-17T16:11:16.940 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-17T16:11:16.940 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-17T16:11:16.940 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-17 16:11:11.653945928 +0000 2024-01-17T16:11:16.941 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-17 16:01:18.083866052 +0000 2024-01-17T16:11:16.941 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-17 16:01:18.083866052 +0000 2024-01-17T16:11:16.941 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-01-17 16:01:18.083866052 +0000 2024-01-17T16:11:16.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-17T16:11:17.004 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-17T16:11:17.004 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-17T16:11:17.004 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.9941e-05 s, 5.7 MB/s 2024-01-17T16:11:17.005 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-17T16:11:17.060 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-17 16:11:11.653945928 +0000 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-17 16:01:18.414859864 +0000 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-17 16:01:18.414859864 +0000 2024-01-17T16:11:17.115 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-01-17 16:01:18.414859864 +0000 2024-01-17T16:11:17.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-17T16:11:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-17T16:11:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-17T16:11:17.180 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.4889e-05 s, 5.4 MB/s 2024-01-17T16:11:17.181 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-17T16:11:17.236 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-01-17 16:11:11.654945910 +0000 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-01-17 16:01:18.756853467 +0000 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-01-17 16:01:18.756853467 +0000 2024-01-17T16:11:17.290 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-01-17 16:01:18.756853467 +0000 2024-01-17T16:11:17.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-17T16:11:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-01-17T16:11:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-01-17T16:11:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.7149e-05 s, 5.9 MB/s 2024-01-17T16:11:17.356 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-17T16:11:17.411 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-17T16:11:17.411 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-17T16:11:17.411 INFO:tasks.ceph:Generating config... 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-17T16:11:17.412 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] err to stderr = True 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] log flush on exit = True 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] log to file = False 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] log to stderr = True 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-17T16:11:17.413 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-17T16:11:17.413 INFO:tasks.ceph:[global] auth client required = none 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] auth cluster required = none 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] auth service required = none 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] auth supported = none 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug asok = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug auth = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug client = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug context = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug crush = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-01-17T16:11:17.414 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug filer = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug journal = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug mds = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-01-17T16:11:17.415 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug mon = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug monc = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug ms = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug osd = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug rados = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-01-17T16:11:17.416 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug refs = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug striper = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug timer = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] debug tp = 0/0 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] ms client mode = crc 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-01-17T16:11:17.417 INFO:tasks.ceph:[global] ms service mode = crc 2024-01-17T16:11:17.417 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-17T16:11:17.417 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-17T16:11:17.418 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-01-17T16:11:17.418 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-01-17T16:11:17.418 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-01-17T16:11:17.418 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-01-17T16:11:17.418 INFO:tasks.ceph:Setting up mon.a... 2024-01-17T16:11:17.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-17T16:11:17.505 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2024-01-17T16:11:17.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-17T16:11:17.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-17T16:11:17.583 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29')] 2024-01-17T16:11:17.583 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-01-17T16:11:17.584 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:17.584 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-17T16:11:17.637 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:17.721+0000 7feed8228e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:17.722+0000 7feed8228e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:17.722+0000 7feed8228e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:17.721 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid 63b99694-16d2-45cb-8b43-1c482e75483e 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:fsid 63b99694-16d2-45cb-8b43-1c482e75483e 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2024-01-17T16:11:17.723577+0000 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-01-17T16:11:17.723577+0000 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 17 (quincy) 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-01-17T16:11:17.722 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-17T16:11:17.723 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-17T16:11:17.737 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 63b99694-16d2-45cb-8b43-1c482e75483e... 2024-01-17T16:11:17.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-17T16:11:17.826 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-01-17T16:11:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = none 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = 172.21.15.29 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: auth client required = none 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: auth cluster required = none 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: auth service required = none 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: debug asok = 0/0 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 0/0 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: debug bdev = 0/0 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 0/0 2024-01-17T16:11:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug buffer = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug civetweb = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug compressor = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug context = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug crush = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug crypto = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug dpdk = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug eventtrace = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug filer = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug filestore = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug finisher = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug fuse = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug heartbeatmap = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug javaclient = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug journal = 0/0 2024-01-17T16:11:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: debug journaler = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug kstore = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug leveldb = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug lockdep = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds balancer = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds locker = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds log = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds log expire = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds migrator = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug memdb = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgrc = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug objclass = 0/0 2024-01-17T16:11:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: debug objectcacher = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug objecter = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug optracker = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug perfcounter = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rados = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd mirror = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rbd replay = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug refs = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rgw = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rgw sync = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug striper = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug throttle = 0/0 2024-01-17T16:11:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: debug timer = 0/0 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: debug tp = 0/0 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: ms client mode = crc 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: ms cluster mode = crc 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: ms mon client mode = crc 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: ms mon cluster mode = crc 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: ms mon service mode = crc 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: ms service mode = crc 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 63b99694-16d2-45cb-8b43-1c482e75483e 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2024-01-17T16:11:17.833 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.834 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression algorithm = snappy 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression mode = aggressive 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore rocksdb cf = False 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: crimson seastar smp = 3 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 20 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 20 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: debug filestore = 0/0 2024-01-17T16:11:17.835 INFO:teuthology.orchestra.run.smithi029.stdout: debug journal = 0/0 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 0/0 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 0/0 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: err to stderr = True 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: log flush on exit = True 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: log to file = False 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: log to stderr = True 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: osd shutdown pgref assert = True 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-01-17T16:11:17.836 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 0/0 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 0/0 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 0/0 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2024-01-17T16:11:17.837 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default crimson = True 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2024-01-17T16:11:17.838 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2024-01-17T16:11:17.839 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2024-01-17T16:11:17.839 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2024-01-17T16:11:17.839 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-17T16:11:17.839 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-17T16:11:17.839 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2024-01-17T16:11:17.840 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-17T16:11:17.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-17T16:11:17.900 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-17T16:11:17.900 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:17.900 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-17T16:11:17.914 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:17.915 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-17T16:11:17.969 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:11:17.969 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:17.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-17T16:11:17.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-17T16:11:18.047 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:18.047 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:18.101 INFO:tasks.ceph:Setting up mon nodes... 2024-01-17T16:11:18.101 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-17T16:11:18.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-17T16:11:18.202 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-17T16:11:18.205 INFO:tasks.ceph:Setting up mds nodes... 2024-01-17T16:11:18.205 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-17T16:11:18.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-17T16:11:18.253 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-17T16:11:18.268 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-17T16:11:18.268 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-17T16:11:18.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-17T16:11:18.294 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-17T16:11:18.294 INFO:tasks.ceph:role: osd.0 2024-01-17T16:11:18.294 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:11:18.294 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-17T16:11:18.360 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-17T16:11:18.360 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-17T16:11:18.360 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-17T16:11:18.360 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-17T16:11:18.360 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-17T16:11:18.360 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-01-17T16:11:18.361 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-17T16:11:18.361 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-17T16:11:18.361 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-17T16:11:18.361 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-17T16:11:18.392 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-01-17T16:11:18.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-01-17T16:11:18.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-17T16:11:18.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-17T16:11:18.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-17T16:11:18.608 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-17T16:11:18.608 INFO:tasks.ceph:role: osd.1 2024-01-17T16:11:18.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:11:18.609 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-01-17T16:11:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-17T16:11:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-17T16:11:18.676 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-17T16:11:18.676 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-17T16:11:18.708 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-01-17T16:11:18.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-01-17T16:11:18.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-17T16:11:18.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-17T16:11:18.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-17T16:11:18.918 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-17T16:11:18.918 INFO:tasks.ceph:role: osd.2 2024-01-17T16:11:18.918 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:11:18.918 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-17T16:11:18.984 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-17T16:11:19.017 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-01-17T16:11:19.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-01-17T16:11:19.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-17T16:11:19.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-17T16:11:19.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-17T16:11:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:19,688 seastar - Reactor backend: linux-aio 2024-01-17T16:11:19.706 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:19,707 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-17T16:11:19.706 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:19,707 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-17T16:11:19.706 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:19,707 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-17T16:11:19.706 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:19,707 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-17T16:11:19.708 INFO:teuthology.orchestra.run.smithi029.stderr:==28105==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-17T16:11:19.714 INFO:teuthology.orchestra.run.smithi029.stderr:==28105==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffceda62000; bottom 0x7f9df82e1000; size: 0x005ef5781000 (407845212160) 2024-01-17T16:11:19.714 INFO:teuthology.orchestra.run.smithi029.stderr:False positive error reports may follow 2024-01-17T16:11:19.714 INFO:teuthology.orchestra.run.smithi029.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-17T16:11:20.260 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-01-17T16:11:20.261 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-01-17T16:11:20.261 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-01-17T16:11:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:20.261 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:20.315 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-17T16:11:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:20,318 seastar - Reactor backend: linux-aio 2024-01-17T16:11:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:20,336 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-17T16:11:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:20,336 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-17T16:11:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:20,336 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-17T16:11:20.342 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:20,343 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-17T16:11:20.344 INFO:teuthology.orchestra.run.smithi029.stderr:==28103==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-17T16:11:20.350 INFO:teuthology.orchestra.run.smithi029.stderr:==28103==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffceda62000; bottom 0x7f9df60ca000; size: 0x005ef7998000 (407880957952) 2024-01-17T16:11:20.351 INFO:teuthology.orchestra.run.smithi029.stderr:False positive error reports may follow 2024-01-17T16:11:20.351 INFO:teuthology.orchestra.run.smithi029.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-17T16:11:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e7e3e3c 0x2e7e51ea 0x2e7e671c 0x2e7e6aa0 0x2e7e6bea 0x2e7e7040 0x54daf 0x1f4c9517 0x23f4c2bd 0x23f50c1c 0x25e7dea2 0x25e84f20 0x25e8786d 0x25e88a32 0x23a93c5d 0x23ab0bc4 0x23b25235 0x2e7567f5 0x2e7b9a26 0x2ea5fbab 0x2ea619f5 0x2e412ca4 0x2e414bad 0x1f6e776c 0x3feaf 0x3ff5f 0x1f2a6a54 2024-01-17T16:11:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-01-17T16:11:24.381 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-01-17T16:11:24.382 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-01-17T16:11:24.382 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-01-17T16:11:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:24.382 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:24.382 INFO:teuthology.orchestra.run.smithi029.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 63b99694-16d2-45cb-8b43-1c482e75483e 2024-01-17T16:11:24.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-17T16:11:24.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:24.975 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-17T16:11:24.976 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:24,978 seastar - Reactor backend: linux-aio 2024-01-17T16:11:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:24,993 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-17T16:11:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:24,993 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-17T16:11:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:24,993 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-17T16:11:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:24,993 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-17T16:11:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:==28287==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-17T16:11:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:==28287==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff49244000; bottom 0x7f41b28cd000; size: 0x00bd96977000 (814275325952) 2024-01-17T16:11:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:False positive error reports may follow 2024-01-17T16:11:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-17T16:11:25.026 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e7e3e3c 0x2e7e51ea 0x2e7e671c 0x2e7e6aa0 0x2e7e6bea 0x2e7e7040 0x54daf 0x24117234 0x2411a953 0x2411ae97 0x24126bba 0x24128244 0x24171986 0x2413c7fc 0x241403fc 0x24140963 0x2416d09d 0x2416d909 0x241211fd 0x2413064a 0x24131d29 0x241326fe 0x241337c4 0x241485fa 0x24148d8b 0x2414acba 0x2414c6be 0x2414d41c 0x2410d88b 0x2410cefa 0x23fb5f13 0x25e7d3ec 0x25e84f20 0x25e8786d 0x25e88a32 0x23a93c5d 0x23ab0bc4 0x23b25235 0x2e7567f5 0x2e7b9a26 0x2ea5fbab 0x2ea619f5 0x2e412ca4 0x2e414bad 0x1f908005 0x1f90eaad 0x1f6e5eb6 0x3feaf 0x3ff5f 0x1f2a6a54 2024-01-17T16:11:25.026 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-01-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-01-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-01-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-01-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-17T16:11:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:25,592 seastar - Reactor backend: linux-aio 2024-01-17T16:11:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:25,610 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-17T16:11:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:25,610 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-17T16:11:25.608 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:25,610 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-17T16:11:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:25,617 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-17T16:11:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:==28285==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-17T16:11:25.624 INFO:teuthology.orchestra.run.smithi029.stderr:==28285==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff49244000; bottom 0x7f41b06b6000; size: 0x00bd98b8e000 (814311071744) 2024-01-17T16:11:25.624 INFO:teuthology.orchestra.run.smithi029.stderr:False positive error reports may follow 2024-01-17T16:11:25.624 INFO:teuthology.orchestra.run.smithi029.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-17T16:11:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e7e3e3c 0x2e7e51ea 0x2e7e671c 0x2e7e6aa0 0x2e7e6bea 0x2e7e7040 0x54daf 0x23df845b 0x23df8891 0x23ead765 0x23eb0975 0x23f463b9 0x23f4b5ab 0x23f50c1c 0x25e7dea2 0x25e84f20 0x25e8786d 0x25e88a32 0x23a93c5d 0x23ab0bc4 0x23b25235 0x2e7567f5 0x2e7b9a26 0x2ea5fbab 0x2ea619f5 0x2e412ca4 0x2e414bad 0x1f6e776c 0x3feaf 0x3ff5f 0x1f2a6a54 2024-01-17T16:11:25.650 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:29.627 INFO:teuthology.orchestra.run.smithi029.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 63b99694-16d2-45cb-8b43-1c482e75483e 2024-01-17T16:11:29.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-17T16:11:29.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:30.216 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-17T16:11:30.217 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,219 seastar - Reactor backend: linux-aio 2024-01-17T16:11:30.238 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,239 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-17T16:11:30.238 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,239 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-17T16:11:30.238 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,239 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-17T16:11:30.238 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,239 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-17T16:11:30.240 INFO:teuthology.orchestra.run.smithi029.stderr:==28469==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-17T16:11:30.246 INFO:teuthology.orchestra.run.smithi029.stderr:==28469==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffea82e5000; bottom 0x7f6d4f0d4000; size: 0x009159211000 (624265596928) 2024-01-17T16:11:30.246 INFO:teuthology.orchestra.run.smithi029.stderr:False positive error reports may follow 2024-01-17T16:11:30.246 INFO:teuthology.orchestra.run.smithi029.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-17T16:11:30.272 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e7e3e3c 0x2e7e51ea 0x2e7e671c 0x2e7e6aa0 0x2e7e6bea 0x2e7e7040 0x54daf 0x2413411b 0x241351f5 0x241362fe 0x24113404 0x24113aa6 0x2411f339 0x24145762 0x24131517 0x241326fe 0x241337c4 0x241485fa 0x24148d8b 0x2414acba 0x2414c6be 0x2414d41c 0x2410d88b 0x2410cefa 0x23fb5f13 0x25e7d3ec 0x25e84f20 0x25e8786d 0x25e88a32 0x23a93c5d 0x23ab0bc4 0x23b25235 0x2e7567f5 0x2e7b9a26 0x2ea5fbab 0x2ea619f5 0x2e412ca4 0x2e414bad 0x1f908005 0x1f90eaad 0x1f6e5eb6 0x3feaf 0x3ff5f 0x1f2a6a54 2024-01-17T16:11:30.272 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-01-17T16:11:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:30.775 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-01-17T16:11:30.775 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-01-17T16:11:30.775 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-01-17T16:11:30.775 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:30.775 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-17T16:11:30.828 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,830 seastar - Reactor backend: linux-aio 2024-01-17T16:11:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,848 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-17T16:11:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,848 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-17T16:11:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,848 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-17T16:11:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:INFO 2024-01-17 16:11:30,856 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-17T16:11:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:==28467==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-17T16:11:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:==28467==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffea82e5000; bottom 0x7f6d4cebd000; size: 0x00915b428000 (624301342720) 2024-01-17T16:11:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:False positive error reports may follow 2024-01-17T16:11:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-17T16:11:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e7e3e3c 0x2e7e51ea 0x2e7e671c 0x2e7e6aa0 0x2e7e6bea 0x2e7e7040 0x54daf 0x23df83e3 0x23df8891 0x23ead765 0x23eb0975 0x23f463b9 0x23f4b5ab 0x23f50c1c 0x25e7dea2 0x25e84f20 0x25e8786d 0x25e88a32 0x23a93c5d 0x23ab0bc4 0x23b25235 0x2e7567f5 0x2e7b9a26 0x2ea5fbab 0x2ea619f5 0x2e412ca4 0x2e414bad 0x1f6e776c 0x3feaf 0x3ff5f 0x1f2a6a54 2024-01-17T16:11:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:kernel callstack: 2024-01-17T16:11:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:Suppressions used: 2024-01-17T16:11:34.898 INFO:teuthology.orchestra.run.smithi029.stderr: count bytes template 2024-01-17T16:11:34.898 INFO:teuthology.orchestra.run.smithi029.stderr: 1 8 MallocExtension::Register 2024-01-17T16:11:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:----------------------------------------------------- 2024-01-17T16:11:34.899 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:34.899 INFO:teuthology.orchestra.run.smithi029.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 63b99694-16d2-45cb-8b43-1c482e75483e 2024-01-17T16:11:34.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-17T16:11:34.966 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-17T16:11:34.966 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:34.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-17T16:11:35.035 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:35.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-17T16:11:35.101 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:35.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-17T16:11:35.169 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:35.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-17T16:11:35.236 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:35.236 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-17T16:11:35.291 INFO:tasks.ceph:Adding keys to all mons... 2024-01-17T16:11:35.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-17T16:11:35.356 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2024-01-17T16:11:35.356 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAm/KdlmBczDBAAudhoyVs4R/W9TabJMFOD9w== 2024-01-17T16:11:35.356 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAo/KdlOy30FxAA4g1bFtAztPjN+4PFClAohw== 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAt/KdlCFhEKBAALoFNhHY3HodcQ8whHDLgtA== 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAy/KdleN1wNhAAB5EkPP9pEKdBRHpmK5gbfw== 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-01-17T16:11:35.357 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAm/KdlLcozDxAArKLmGH3sqdr0H42K+paT5Q== 2024-01-17T16:11:35.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-17T16:11:35.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-17T16:11:35.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-17T16:11:35.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-17T16:11:35.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-17T16:11:35.689 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-17T16:11:35.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-17T16:11:35.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-17T16:11:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:35.813+0000 7f1555c509c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:35.813+0000 7f1555c509c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:35.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-17T16:11:35.852 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-17T16:11:35.907 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-17T16:11:35.908 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-17T16:11:35.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-17T16:11:35.951 INFO:tasks.ceph.mon.a:Started 2024-01-17T16:11:35.951 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-17T16:11:35.951 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-17T16:11:35.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-17T16:11:35.954 INFO:tasks.ceph.mgr.x:Started 2024-01-17T16:11:35.954 INFO:tasks.ceph:Setting crush tunables to default 2024-01-17T16:11:35.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2024-01-17T16:11:36.031 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-01-17T16:11:36.032+0000 7f77bbc519c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.031 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-01-17T16:11:36.032+0000 7f77bbc519c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.031 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-01-17T16:11:36.032+0000 7f77bbc519c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.033 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:36.033+0000 7fcb05a1ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:36.059+0000 7f233fd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:36.076+0000 7f233fd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.146 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-01-17T16:11:36.157 INFO:tasks.ceph:check_enable_crimson: True 2024-01-17T16:11:36.157 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-01-17T16:11:36.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-01-17T16:11:36.236 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:36.236+0000 7fcb05a1ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.236 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:36.237+0000 7fcb05a1ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:36.260+0000 7f34e3d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:36.266+0000 7f34e3d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:36.345 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-01-17T16:11:36.346+0000 7f77b2ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-01-17T16:11:36.345 INFO:tasks.ceph.mon.a.smithi029.stderr:Please be aware that this feature is experimental, untested, 2024-01-17T16:11:36.345 INFO:tasks.ceph.mon.a.smithi029.stderr:unsupported, and may result in data corruption, data loss, 2024-01-17T16:11:36.345 INFO:tasks.ceph.mon.a.smithi029.stderr:and/or irreparable damage to your cluster. Do not use 2024-01-17T16:11:36.345 INFO:tasks.ceph.mon.a.smithi029.stderr:feature with important data. 2024-01-17T16:11:36.345 INFO:tasks.ceph.mon.a.smithi029.stderr: 2024-01-17T16:11:36.708 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:36.708+0000 7fcb05a1ff00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-17T16:11:36.762 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:36.763+0000 7fcb05a1ff00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-17T16:11:37.158 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-17T16:11:37.159 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:37.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-17T16:11:37.186 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:37.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-17T16:11:37.253 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:37.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-17T16:11:37.319 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.319+0000 7fcb05a1ff00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-17T16:11:37.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd new 253695fd-9058-42b6-85f3-07d6f2f62329 0 2024-01-17T16:11:37.373 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.374+0000 7fcb05a1ff00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-17T16:11:37.420 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.421+0000 7fcb05a1ff00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-17T16:11:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:37.462+0000 7f8d3e617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:37.469+0000 7f8d3e617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:37.478 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.479+0000 7fcb05a1ff00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-17T16:11:37.522 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.523+0000 7fcb05a1ff00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-17T16:11:37.546 INFO:teuthology.orchestra.run.smithi029.stdout:0 2024-01-17T16:11:37.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd new 42a7b703-3995-49be-b2cf-bf6a8d8117a8 1 2024-01-17T16:11:37.629 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.629+0000 7fcb05a1ff00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-17T16:11:37.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:37.698+0000 7f3776e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:37.703 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.704+0000 7fcb05a1ff00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-17T16:11:37.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:37.705+0000 7f3776e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:37.757 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.757+0000 7fcb05a1ff00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-17T16:11:37.789 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-01-17T16:11:37.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd new fe5b7cb6-bc60-44ce-8f92-d73124581e00 2 2024-01-17T16:11:37.811 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.812+0000 7fcb05a1ff00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-17T16:11:37.861 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:37.862+0000 7fcb05a1ff00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-17T16:11:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:37.913+0000 7f62a415e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:37.919+0000 7f62a415e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:37.997 INFO:teuthology.orchestra.run.smithi029.stdout:2 2024-01-17T16:11:38.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-01-17T16:11:38.039 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-17T16:11:38.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-01-17T16:11:38.082 INFO:tasks.ceph.osd.0:Started 2024-01-17T16:11:38.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-01-17T16:11:38.113 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-17T16:11:38.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-01-17T16:11:38.155 INFO:tasks.ceph.osd.1:Started 2024-01-17T16:11:38.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-01-17T16:11:38.193 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-17T16:11:38.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-01-17T16:11:38.237 INFO:tasks.ceph.osd.2:Started 2024-01-17T16:11:38.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-17T16:11:38.291 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:38.291+0000 7fcb05a1ff00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-17T16:11:38.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:38.430+0000 7fe5cc66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:38.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:38.440+0000 7fe5cc66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:38.466 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:38.466+0000 7fcb05a1ff00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-17T16:11:38.579 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:38.579 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":6,"fsid":"63b99694-16d2-45cb-8b43-1c482e75483e","created":"2024-01-17T16:11:36.071150+0000","modified":"2024-01-17T16:11:37.998347+0000","last_up_change":"0.000000","last_in_change":"2024-01-17T16:11:37.998347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"253695fd-9058-42b6-85f3-07d6f2f62329","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42a7b703-3995-49be-b2cf-bf6a8d8117a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fe5b7cb6-bc60-44ce-8f92-d73124581e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-17T16:11:38.599 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-17T16:11:38.599 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-17T16:11:38.604 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:38.605+0000 7fcb05a1ff00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-17T16:11:38.657 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:38.658+0000 7fcb05a1ff00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-17T16:11:38.771 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:38.772+0000 7fcb05a1ff00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-17T16:11:38.871 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:38.872+0000 7fcb05a1ff00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-17T16:11:38.901 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-17T16:11:39.000 INFO:teuthology.misc.health.smithi029.stderr:2024-01-17T16:11:38.997+0000 7f5bc646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:39.002 INFO:teuthology.misc.health.smithi029.stderr:2024-01-17T16:11:39.003+0000 7f5bc646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:39.025 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:39.026+0000 7fcb05a1ff00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-17T16:11:39.069 INFO:teuthology.misc.health.smithi029.stdout: 2024-01-17T16:11:39.069 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":6,"fsid":"63b99694-16d2-45cb-8b43-1c482e75483e","created":"2024-01-17T16:11:36.071150+0000","modified":"2024-01-17T16:11:37.998347+0000","last_up_change":"0.000000","last_in_change":"2024-01-17T16:11:37.998347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"253695fd-9058-42b6-85f3-07d6f2f62329","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"42a7b703-3995-49be-b2cf-bf6a8d8117a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fe5b7cb6-bc60-44ce-8f92-d73124581e00","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-17T16:11:39.072 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:39.073+0000 7fcb05a1ff00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-17T16:11:39.080 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-17T16:11:39.226 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:39.227+0000 7fcb05a1ff00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-17T16:11:40.116 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:40.116+0000 7fcb05a1ff00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-17T16:11:40.422 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:40.423+0000 7fcb05a1ff00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-17T16:11:40.476 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:40.477+0000 7fcb05a1ff00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-17T16:11:40.585 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:40.586+0000 7fcb05a1ff00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-17T16:11:41.605 INFO:tasks.ceph.mgr.x.smithi029.stderr:2024-01-17T16:11:41.605+0000 7fcaa5c11640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-17T16:11:45.383 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-17T16:11:45.491 INFO:teuthology.misc.health.smithi029.stderr:2024-01-17T16:11:45.489+0000 7fc455d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:45.493 INFO:teuthology.misc.health.smithi029.stderr:2024-01-17T16:11:45.494+0000 7fc455d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:45.684 INFO:teuthology.misc.health.smithi029.stdout: 2024-01-17T16:11:45.685 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":9,"fsid":"63b99694-16d2-45cb-8b43-1c482e75483e","created":"2024-01-17T16:11:36.071150+0000","modified":"2024-01-17T16:11:45.611523+0000","last_up_change":"2024-01-17T16:11:43.606617+0000","last_in_change":"2024-01-17T16:11:37.998347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-17T16:11:44.646119+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"253695fd-9058-42b6-85f3-07d6f2f62329","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3265042504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":3265042504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3265042504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":3265042504}]},"public_addr":"172.21.15.29:6802/3265042504","cluster_addr":"172.21.15.29:6803/3265042504","heartbeat_back_addr":"172.21.15.29:6808/3265042504","heartbeat_front_addr":"172.21.15.29:6811/3265042504","state":["exists","up"]},{"osd":1,"uuid":"42a7b703-3995-49be-b2cf-bf6a8d8117a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2097608913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2097608913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2097608913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":2097608913}]},"public_addr":"172.21.15.29:6806/2097608913","cluster_addr":"172.21.15.29:6807/2097608913","heartbeat_back_addr":"172.21.15.29:6812/2097608913","heartbeat_front_addr":"172.21.15.29:6813/2097608913","state":["exists","up"]},{"osd":2,"uuid":"fe5b7cb6-bc60-44ce-8f92-d73124581e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":978500373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":978500373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":978500373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":978500373}]},"public_addr":"172.21.15.29:6804/978500373","cluster_addr":"172.21.15.29:6805/978500373","heartbeat_back_addr":"172.21.15.29:6809/978500373","heartbeat_front_addr":"172.21.15.29:6810/978500373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-17T16:11:45.696 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-17T16:11:45.696 INFO:tasks.ceph:Creating RBD pool 2024-01-17T16:11:45.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-17T16:11:45.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:45.797+0000 7fecb555e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:45.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:45.805+0000 7fecb555e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:46.612 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2024-01-17T16:11:46.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-17T16:11:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:46.740+0000 7fdd21515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:46.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:46.751+0000 7fdd21515640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-17T16:11:47.632 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-17T16:11:47.632 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-17T16:11:47.633 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-17T16:11:47.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-17T16:11:47.633 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-17T16:11:47.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:47.759+0000 7fa43855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:47.764+0000 7fa43855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.001 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:48.012 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":4180030409},{"type":"v1","addr":"172.21.15.29:6801","nonce":4180030409}]},"active_addr":"172.21.15.29:6801/4180030409","active_change":"2024-01-17T16:11:40.600103+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3806980968}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":994130748}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2542156253}]}]} 2024-01-17T16:11:48.014 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-17T16:11:48.015 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-17T16:11:48.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-17T16:11:48.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.119+0000 7f70b726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.124+0000 7f70b726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.311 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:48.312 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":11,"fsid":"63b99694-16d2-45cb-8b43-1c482e75483e","created":"2024-01-17T16:11:36.071150+0000","modified":"2024-01-17T16:11:47.617578+0000","last_up_change":"2024-01-17T16:11:43.606617+0000","last_in_change":"2024-01-17T16:11:37.998347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-17T16:11:44.646119+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-17T16:11:46.005775+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"253695fd-9058-42b6-85f3-07d6f2f62329","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3265042504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":3265042504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3265042504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":3265042504}]},"public_addr":"172.21.15.29:6802/3265042504","cluster_addr":"172.21.15.29:6803/3265042504","heartbeat_back_addr":"172.21.15.29:6808/3265042504","heartbeat_front_addr":"172.21.15.29:6811/3265042504","state":["exists","up"]},{"osd":1,"uuid":"42a7b703-3995-49be-b2cf-bf6a8d8117a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2097608913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2097608913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2097608913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":2097608913}]},"public_addr":"172.21.15.29:6806/2097608913","cluster_addr":"172.21.15.29:6807/2097608913","heartbeat_back_addr":"172.21.15.29:6812/2097608913","heartbeat_front_addr":"172.21.15.29:6813/2097608913","state":["exists","up"]},{"osd":2,"uuid":"fe5b7cb6-bc60-44ce-8f92-d73124581e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":978500373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":978500373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":978500373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":978500373}]},"public_addr":"172.21.15.29:6804/978500373","cluster_addr":"172.21.15.29:6805/978500373","heartbeat_back_addr":"172.21.15.29:6809/978500373","heartbeat_front_addr":"172.21.15.29:6810/978500373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-17T16:11:48.324 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-17T16:11:48.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-17T16:11:48.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.429+0000 7f0cc4d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.434+0000 7f0cc4d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.624 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:48.625 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"fsid":"63b99694-16d2-45cb-8b43-1c482e75483e","created":"2024-01-17T16:11:36.071150+0000","modified":"2024-01-17T16:11:48.620065+0000","last_up_change":"2024-01-17T16:11:43.606617+0000","last_in_change":"2024-01-17T16:11:37.998347+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-17T16:11:44.646119+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-17T16:11:46.005775+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"253695fd-9058-42b6-85f3-07d6f2f62329","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3265042504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":3265042504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3265042504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":3265042504}]},"public_addr":"172.21.15.29:6802/3265042504","cluster_addr":"172.21.15.29:6803/3265042504","heartbeat_back_addr":"172.21.15.29:6808/3265042504","heartbeat_front_addr":"172.21.15.29:6811/3265042504","state":["exists","up"]},{"osd":1,"uuid":"42a7b703-3995-49be-b2cf-bf6a8d8117a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2097608913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2097608913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":2097608913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":2097608913}]},"public_addr":"172.21.15.29:6806/2097608913","cluster_addr":"172.21.15.29:6807/2097608913","heartbeat_back_addr":"172.21.15.29:6812/2097608913","heartbeat_front_addr":"172.21.15.29:6813/2097608913","state":["exists","up"]},{"osd":2,"uuid":"fe5b7cb6-bc60-44ce-8f92-d73124581e00","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":978500373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":978500373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":978500373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":978500373}]},"public_addr":"172.21.15.29:6804/978500373","cluster_addr":"172.21.15.29:6805/978500373","heartbeat_back_addr":"172.21.15.29:6809/978500373","heartbeat_front_addr":"172.21.15.29:6810/978500373","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-17T16:11:48.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-17T16:11:48.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-17T16:11:48.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-17T16:11:48.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.754+0000 7f5e54f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.754+0000 7f5d3ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.754+0000 7fa13f99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.761+0000 7f5e54f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.761+0000 7f5d3ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.761+0000 7fa13f99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.782 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2024-01-17T16:11:48.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-17T16:11:48.789 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2024-01-17T16:11:48.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-17T16:11:48.794 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2024-01-17T16:11:48.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-17T16:11:48.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.889+0000 7fa57fd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.895+0000 7fa57fd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.942+0000 7f6d6b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.942+0000 7fba2926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.948+0000 7fba2926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:48.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:48.949+0000 7f6d6b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2024-01-17T16:11:49.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2024-01-17T16:11:49.122 DEBUG:teuthology.parallel:result is None 2024-01-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2024-01-17T16:11:49.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2024-01-17T16:11:49.148 DEBUG:teuthology.parallel:result is None 2024-01-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi029.stdout:34359738373 2024-01-17T16:11:49.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2024-01-17T16:11:49.176 DEBUG:teuthology.parallel:result is None 2024-01-17T16:11:49.176 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-17T16:11:49.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-17T16:11:49.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:49.321+0000 7f238a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:49.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:49.326+0000 7f238a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:49.508 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:49.509 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-01-17T16:11:49.519 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-01-17T16:11:48.657926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":421888,"data_stored":1251088,"data_compressed":59834,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.647086+0000","last_change":"2024-01-17T16:11:47.732449+0000","last_active":"2024-01-17T16:11:48.647086+0000","last_peered":"2024-01-17T16:11:48.647086+0000","last_clean":"2024-01-17T16:11:48.647086+0000","last_became_active":"2024-01-17T16:11:47.727969+0000","last_became_peered":"2024-01-17T16:11:47.727969+0000","last_unstale":"2024-01-17T16:11:48.647086+0000","last_undegraded":"2024-01-17T16:11:48.647086+0000","last_fullsized":"2024-01-17T16:11:48.647086+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.645219+0000","last_change":"2024-01-17T16:11:47.694542+0000","last_active":"2024-01-17T16:11:48.645219+0000","last_peered":"2024-01-17T16:11:48.645219+0000","last_clean":"2024-01-17T16:11:48.645219+0000","last_became_active":"2024-01-17T16:11:47.691960+0000","last_became_peered":"2024-01-17T16:11:47.691960+0000","last_unstale":"2024-01-17T16:11:48.645219+0000","last_undegraded":"2024-01-17T16:11:48.645219+0000","last_fullsized":"2024-01-17T16:11:48.645219+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.645923+0000","last_change":"2024-01-17T16:11:47.690618+0000","last_active":"2024-01-17T16:11:48.645923+0000","last_peered":"2024-01-17T16:11:48.645923+0000","last_clean":"2024-01-17T16:11:48.645923+0000","last_became_active":"2024-01-17T16:11:47.682844+0000","last_became_peered":"2024-01-17T16:11:47.682844+0000","last_unstale":"2024-01-17T16:11:48.645923+0000","last_undegraded":"2024-01-17T16:11:48.645923+0000","last_fullsized":"2024-01-17T16:11:48.645923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.648331+0000","last_change":"2024-01-17T16:11:47.721593+0000","last_active":"2024-01-17T16:11:48.648331+0000","last_peered":"2024-01-17T16:11:48.648331+0000","last_clean":"2024-01-17T16:11:48.648331+0000","last_became_active":"2024-01-17T16:11:47.702679+0000","last_became_peered":"2024-01-17T16:11:47.702679+0000","last_unstale":"2024-01-17T16:11:48.648331+0000","last_undegraded":"2024-01-17T16:11:48.648331+0000","last_fullsized":"2024-01-17T16:11:48.648331+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-17T16:11:47.710717+0000","last_change":"2024-01-17T16:11:47.710802+0000","last_active":"2024-01-17T16:11:47.710717+0000","last_peered":"2024-01-17T16:11:47.710717+0000","last_clean":"2024-01-17T16:11:46.613944+0000","last_became_active":"2024-01-17T16:11:47.708546+0000","last_became_peered":"2024-01-17T16:11:47.708546+0000","last_unstale":"2024-01-17T16:11:47.710717+0000","last_undegraded":"2024-01-17T16:11:47.710717+0000","last_fullsized":"2024-01-17T16:11:47.710717+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.636639+0000","last_change":"2024-01-17T16:11:47.723083+0000","last_active":"2024-01-17T16:11:48.636639+0000","last_peered":"2024-01-17T16:11:48.636639+0000","last_clean":"2024-01-17T16:11:48.636639+0000","last_became_active":"2024-01-17T16:11:47.720461+0000","last_became_peered":"2024-01-17T16:11:47.720461+0000","last_unstale":"2024-01-17T16:11:48.636639+0000","last_undegraded":"2024-01-17T16:11:48.636639+0000","last_fullsized":"2024-01-17T16:11:48.636639+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.636507+0000","last_change":"2024-01-17T16:11:47.718643+0000","last_active":"2024-01-17T16:11:48.636507+0000","last_peered":"2024-01-17T16:11:48.636507+0000","last_clean":"2024-01-17T16:11:48.636507+0000","last_became_active":"2024-01-17T16:11:47.715773+0000","last_became_peered":"2024-01-17T16:11:47.715773+0000","last_unstale":"2024-01-17T16:11:48.636507+0000","last_undegraded":"2024-01-17T16:11:48.636507+0000","last_fullsized":"2024-01-17T16:11:48.636507+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.646685+0000","last_change":"2024-01-17T16:11:47.695824+0000","last_active":"2024-01-17T16:11:48.646685+0000","last_peered":"2024-01-17T16:11:48.646685+0000","last_clean":"2024-01-17T16:11:48.646685+0000","last_became_active":"2024-01-17T16:11:47.693321+0000","last_became_peered":"2024-01-17T16:11:47.693321+0000","last_unstale":"2024-01-17T16:11:48.646685+0000","last_undegraded":"2024-01-17T16:11:48.646685+0000","last_fullsized":"2024-01-17T16:11:48.646685+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.644330+0000","last_change":"2024-01-17T16:11:46.674963+0000","last_active":"2024-01-17T16:11:48.644330+0000","last_peered":"2024-01-17T16:11:48.644330+0000","last_clean":"2024-01-17T16:11:48.644330+0000","last_became_active":"2024-01-17T16:11:46.670641+0000","last_became_peered":"2024-01-17T16:11:46.670641+0000","last_unstale":"2024-01-17T16:11:48.644330+0000","last_undegraded":"2024-01-17T16:11:48.644330+0000","last_fullsized":"2024-01-17T16:11:48.644330+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:45.611523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:45.611523+0000","last_clean_scrub_stamp":"2024-01-17T16:11:45.611523+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614783,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-17T16:11:49.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-17T16:11:49.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:49.626+0000 7f05ebd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:49.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:49.632+0000 7f05ebd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:49.813 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:49.814 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-01-17T16:11:49.825 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-01-17T16:11:48.657926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":421888,"data_stored":1251088,"data_compressed":59834,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.647086+0000","last_change":"2024-01-17T16:11:47.732449+0000","last_active":"2024-01-17T16:11:48.647086+0000","last_peered":"2024-01-17T16:11:48.647086+0000","last_clean":"2024-01-17T16:11:48.647086+0000","last_became_active":"2024-01-17T16:11:47.727969+0000","last_became_peered":"2024-01-17T16:11:47.727969+0000","last_unstale":"2024-01-17T16:11:48.647086+0000","last_undegraded":"2024-01-17T16:11:48.647086+0000","last_fullsized":"2024-01-17T16:11:48.647086+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.645219+0000","last_change":"2024-01-17T16:11:47.694542+0000","last_active":"2024-01-17T16:11:48.645219+0000","last_peered":"2024-01-17T16:11:48.645219+0000","last_clean":"2024-01-17T16:11:48.645219+0000","last_became_active":"2024-01-17T16:11:47.691960+0000","last_became_peered":"2024-01-17T16:11:47.691960+0000","last_unstale":"2024-01-17T16:11:48.645219+0000","last_undegraded":"2024-01-17T16:11:48.645219+0000","last_fullsized":"2024-01-17T16:11:48.645219+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.645923+0000","last_change":"2024-01-17T16:11:47.690618+0000","last_active":"2024-01-17T16:11:48.645923+0000","last_peered":"2024-01-17T16:11:48.645923+0000","last_clean":"2024-01-17T16:11:48.645923+0000","last_became_active":"2024-01-17T16:11:47.682844+0000","last_became_peered":"2024-01-17T16:11:47.682844+0000","last_unstale":"2024-01-17T16:11:48.645923+0000","last_undegraded":"2024-01-17T16:11:48.645923+0000","last_fullsized":"2024-01-17T16:11:48.645923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.648331+0000","last_change":"2024-01-17T16:11:47.721593+0000","last_active":"2024-01-17T16:11:48.648331+0000","last_peered":"2024-01-17T16:11:48.648331+0000","last_clean":"2024-01-17T16:11:48.648331+0000","last_became_active":"2024-01-17T16:11:47.702679+0000","last_became_peered":"2024-01-17T16:11:47.702679+0000","last_unstale":"2024-01-17T16:11:48.648331+0000","last_undegraded":"2024-01-17T16:11:48.648331+0000","last_fullsized":"2024-01-17T16:11:48.648331+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-01-17T16:11:47.710717+0000","last_change":"2024-01-17T16:11:47.710802+0000","last_active":"2024-01-17T16:11:47.710717+0000","last_peered":"2024-01-17T16:11:47.710717+0000","last_clean":"2024-01-17T16:11:46.613944+0000","last_became_active":"2024-01-17T16:11:47.708546+0000","last_became_peered":"2024-01-17T16:11:47.708546+0000","last_unstale":"2024-01-17T16:11:47.710717+0000","last_undegraded":"2024-01-17T16:11:47.710717+0000","last_fullsized":"2024-01-17T16:11:47.710717+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.636639+0000","last_change":"2024-01-17T16:11:47.723083+0000","last_active":"2024-01-17T16:11:48.636639+0000","last_peered":"2024-01-17T16:11:48.636639+0000","last_clean":"2024-01-17T16:11:48.636639+0000","last_became_active":"2024-01-17T16:11:47.720461+0000","last_became_peered":"2024-01-17T16:11:47.720461+0000","last_unstale":"2024-01-17T16:11:48.636639+0000","last_undegraded":"2024-01-17T16:11:48.636639+0000","last_fullsized":"2024-01-17T16:11:48.636639+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.636507+0000","last_change":"2024-01-17T16:11:47.718643+0000","last_active":"2024-01-17T16:11:48.636507+0000","last_peered":"2024-01-17T16:11:48.636507+0000","last_clean":"2024-01-17T16:11:48.636507+0000","last_became_active":"2024-01-17T16:11:47.715773+0000","last_became_peered":"2024-01-17T16:11:47.715773+0000","last_unstale":"2024-01-17T16:11:48.636507+0000","last_undegraded":"2024-01-17T16:11:48.636507+0000","last_fullsized":"2024-01-17T16:11:48.636507+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.646685+0000","last_change":"2024-01-17T16:11:47.695824+0000","last_active":"2024-01-17T16:11:48.646685+0000","last_peered":"2024-01-17T16:11:48.646685+0000","last_clean":"2024-01-17T16:11:48.646685+0000","last_became_active":"2024-01-17T16:11:47.693321+0000","last_became_peered":"2024-01-17T16:11:47.693321+0000","last_unstale":"2024-01-17T16:11:48.646685+0000","last_undegraded":"2024-01-17T16:11:48.646685+0000","last_fullsized":"2024-01-17T16:11:48.646685+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:46.613944+0000","last_clean_scrub_stamp":"2024-01-17T16:11:46.613944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-17T16:11:48.644330+0000","last_change":"2024-01-17T16:11:46.674963+0000","last_active":"2024-01-17T16:11:48.644330+0000","last_peered":"2024-01-17T16:11:48.644330+0000","last_clean":"2024-01-17T16:11:48.644330+0000","last_became_active":"2024-01-17T16:11:46.670641+0000","last_became_peered":"2024-01-17T16:11:46.670641+0000","last_unstale":"2024-01-17T16:11:48.644330+0000","last_undegraded":"2024-01-17T16:11:48.644330+0000","last_fullsized":"2024-01-17T16:11:48.644330+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-17T16:11:45.611523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-17T16:11:45.611523+0000","last_clean_scrub_stamp":"2024-01-17T16:11:45.611523+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614783,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29917,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-17T16:11:49.826 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-17T16:11:49.826 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-17T16:11:49.826 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-17T16:11:49.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-17T16:11:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:49.932+0000 7f678739e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:11:49.936+0000 7f678739e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:11:50.160 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:50.160 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-17T16:11:50.173 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-17T16:11:50.173 INFO:teuthology.run_tasks:Running task cbt... 2024-01-17T16:11:50.188 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi029.front.sepia.ceph.com', 'osds': ['ubuntu@smithi029.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi029.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi029.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi029.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-01-17T16:11:50.188 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-01-17T16:11:50.206 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:11:50.206 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-17T16:11:50.258 DEBUG:teuthology.orchestra.run.smithi029:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-01-17T16:11:50.313 INFO:teuthology.orchestra.run.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-01-17T16:11:50.755 INFO:tasks.cbt:Installing collectl 2024-01-17T16:11:50.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-01-17T16:11:50.857 INFO:teuthology.orchestra.run.smithi029.stderr:--2024-01-17 16:11:50-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-01-17T16:11:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-01-17T16:11:51.054 INFO:teuthology.orchestra.run.smithi029.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-01-17T16:11:51.495 INFO:teuthology.orchestra.run.smithi029.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-17T16:11:51.495 INFO:teuthology.orchestra.run.smithi029.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlp_xHAUKatGaDFzUnDGS6QBAdwfIH5gMjIMzkuh27tOChfdy1K7bE_wWaCZn2kjiayJNyr-Ubud3vEVsAwCM1jTLW9A%3D%3D&use_mirror=versaweb&r= [following] 2024-01-17T16:11:51.495 INFO:teuthology.orchestra.run.smithi029.stderr:--2024-01-17 16:11:51-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlp_xHAUKatGaDFzUnDGS6QBAdwfIH5gMjIMzkuh27tOChfdy1K7bE_wWaCZn2kjiayJNyr-Ubud3vEVsAwCM1jTLW9A%3D%3D&use_mirror=versaweb&r= 2024-01-17T16:11:51.495 INFO:teuthology.orchestra.run.smithi029.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-01-17T16:11:51.573 INFO:teuthology.orchestra.run.smithi029.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-01-17T16:11:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-17T16:11:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:Location: https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-01-17T16:11:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:--2024-01-17 16:11:51-- https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-01-17T16:11:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:Resolving versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)... 162.251.232.173 2024-01-17T16:11:52.016 INFO:teuthology.orchestra.run.smithi029.stderr:Connecting to versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)|162.251.232.173|:443... connected. 2024-01-17T16:11:52.408 INFO:teuthology.orchestra.run.smithi029.stderr:HTTP request sent, awaiting response... 200 OK 2024-01-17T16:11:52.409 INFO:teuthology.orchestra.run.smithi029.stderr:Length: 620693 (606K) [application/x-gzip] 2024-01-17T16:11:52.409 INFO:teuthology.orchestra.run.smithi029.stderr:Saving to: ‘collectl.tar.gz’ 2024-01-17T16:11:52.409 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:52.490 INFO:teuthology.orchestra.run.smithi029.stderr: 0K .......... .......... .......... .......... .......... 8% 606K 1s 2024-01-17T16:11:52.554 INFO:teuthology.orchestra.run.smithi029.stderr: 50K .......... .......... .......... .......... .......... 16% 785K 1s 2024-01-17T16:11:52.583 INFO:teuthology.orchestra.run.smithi029.stderr: 100K .......... .......... .......... .......... .......... 24% 1.70M 1s 2024-01-17T16:11:52.620 INFO:teuthology.orchestra.run.smithi029.stderr: 150K .......... .......... .......... .......... .......... 32% 1.30M 0s 2024-01-17T16:11:52.630 INFO:teuthology.orchestra.run.smithi029.stderr: 200K .......... .......... .......... .......... .......... 41% 5.46M 0s 2024-01-17T16:11:52.642 INFO:teuthology.orchestra.run.smithi029.stderr: 250K .......... .......... .......... .......... .......... 49% 3.82M 0s 2024-01-17T16:11:52.655 INFO:teuthology.orchestra.run.smithi029.stderr: 300K .......... .......... .......... .......... .......... 57% 3.72M 0s 2024-01-17T16:11:52.692 INFO:teuthology.orchestra.run.smithi029.stderr: 350K .......... .......... .......... .......... .......... 65% 1.34M 0s 2024-01-17T16:11:52.698 INFO:teuthology.orchestra.run.smithi029.stderr: 400K .......... .......... .......... .......... .......... 74% 7.76M 0s 2024-01-17T16:11:52.702 INFO:teuthology.orchestra.run.smithi029.stderr: 450K .......... .......... .......... .......... .......... 82% 12.4M 0s 2024-01-17T16:11:52.703 INFO:teuthology.orchestra.run.smithi029.stderr: 500K .......... .......... .......... .......... .......... 90% 34.0M 0s 2024-01-17T16:11:52.709 INFO:teuthology.orchestra.run.smithi029.stderr: 550K .......... .......... .......... .......... .......... 98% 8.80M 0s 2024-01-17T16:11:52.710 INFO:teuthology.orchestra.run.smithi029.stderr: 600K ...... 100% 11.4T=0.3s 2024-01-17T16:11:52.710 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:52.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17 16:11:52 (1.97 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-01-17T16:11:52.710 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/ 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/collectl.conf 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/ 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/SlowProc.html 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/WhySummary.html 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Socket.html 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/colmux.html 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Tutorial.html 2024-01-17T16:11:52.750 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Gexpr.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Import.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/style.css 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/FileNaming.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/TheMath.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Environmental.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Process.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/HiResTime.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/slub.jpg 2024-01-17T16:11:52.751 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Lustre.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Documentation.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Inodes.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Graphite.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/InputFiles.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Export.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Interrupts.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Startup.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Performance.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Messages.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Logging.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data-detail.html 2024-01-17T16:11:52.752 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/CPUs.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Matrix.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data-brief.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Misc.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Examples.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OpenStack.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Data.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Exceptions.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-01-17T16:11:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Playback.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Hello.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Sockets.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Network.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Infiniband.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Memory.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/Disks.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/service/ 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/service/collectl.service 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/ 2024-01-17T16:11:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl-suse 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl-debian 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl-generic 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/initd/collectl 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/misc.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/gexpr.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/statsd.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/envrules.std 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/hello.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/vmsum.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/proctree.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/README-WINDOWS 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/formatit.ph 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/colmux 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/INSTALL 2024-01-17T16:11:52.755 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/graphite.ph 2024-01-17T16:11:52.756 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/RELEASE-collectl 2024-01-17T16:11:52.757 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/vmstat.ph 2024-01-17T16:11:52.757 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/README 2024-01-17T16:11:52.757 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/man1/ 2024-01-17T16:11:52.757 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/man1/collectl.1 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/man1/colmux.1 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/lexpr.ph 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/client.pl 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/collectl 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/COPYING 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/vnet.ph 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/GPL 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/UNINSTALL 2024-01-17T16:11:52.758 INFO:teuthology.orchestra.run.smithi029.stdout:collectl-4.3.1/ARTISTIC 2024-01-17T16:11:52.808 INFO:teuthology.orchestra.run.smithi029.stderr:./INSTALL: line 90: chkconfig: command not found 2024-01-17T16:11:52.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-01-17T16:11:53.379 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:07:52 ago on Wed 17 Jan 2024 04:04:01 PM UTC. 2024-01-17T16:11:53.483 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-01-17T16:11:53.486 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-01-17T16:11:53.545 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-01-17T16:11:53.546 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:53.546 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-01-17T16:11:53.546 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:53.546 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-01-17T16:11:53.546 INFO:teuthology.orchestra.run.smithi029.stdout: librbd-devel x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 23 k 2024-01-17T16:11:53.546 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout: perf x86_64 5.14.0-408.el9 CentOS-AppStream 8.0 M 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout: libradospp-devel x86_64 2:19.0.0-720.gcecb6cb5.el9 ceph 30 k 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout:Install 5 Packages 2024-01-17T16:11:53.547 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:53.548 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 8.1 M 2024-01-17T16:11:53.548 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 11 M 2024-01-17T16:11:53.548 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-01-17T16:11:53.971 INFO:teuthology.orchestra.run.smithi029.stdout:(1/5): librbd-devel-19.0.0-720.gcecb6cb5.el9.x8 62 kB/s | 23 kB 00:00 2024-01-17T16:11:54.146 INFO:teuthology.orchestra.run.smithi029.stdout:(2/5): perf-5.14.0-408.el9.x86_64.rpm 15 MB/s | 8.0 MB 00:00 2024-01-17T16:11:54.172 INFO:teuthology.orchestra.run.smithi029.stdout:(3/5): libradospp-devel-19.0.0-720.gcecb6cb5.el 53 kB/s | 30 kB 00:00 2024-01-17T16:11:54.381 INFO:teuthology.orchestra.run.smithi029.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 56 kB/s | 13 kB 00:00 2024-01-17T16:11:54.406 INFO:teuthology.orchestra.run.smithi029.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 273 kB/s | 118 kB 00:00 2024-01-17T16:11:54.409 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-01-17T16:11:54.409 INFO:teuthology.orchestra.run.smithi029.stdout:Total 9.4 MB/s | 8.1 MB 00:00 2024-01-17T16:11:54.410 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-01-17T16:11:54.431 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-01-17T16:11:54.431 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-01-17T16:11:54.610 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-01-17T16:11:54.610 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-01-17T16:11:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-01-17T16:11:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-01-17T16:11:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-01-17T16:11:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradospp-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 3/5 2024-01-17T16:11:55.477 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librbd-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 4/5 2024-01-17T16:11:55.650 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : perf-5.14.0-408.el9.x86_64 5/5 2024-01-17T16:11:57.036 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: perf-5.14.0-408.el9.x86_64 5/5 2024-01-17T16:11:57.037 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradospp-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 1/5 2024-01-17T16:11:57.037 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 2/5 2024-01-17T16:11:57.037 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : perf-5.14.0-408.el9.x86_64 3/5 2024-01-17T16:11:57.037 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-01-17T16:11:57.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: libradospp-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: librbd-devel-2:19.0.0-720.gcecb6cb5.el9.x86_64 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh-2.34-7.el9.x86_64 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: perf-5.14.0-408.el9.x86_64 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-01-17T16:11:57.480 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-01-17T16:11:57.968 INFO:tasks.cbt:benchmark: radosbench 2024-01-17T16:11:57.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-01-17T16:11:57.997 DEBUG:teuthology.orchestra.run.smithi029:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-17T16:11:58.390 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - Settings.cluster: 2024-01-17T16:11:58.390 INFO:teuthology.orchestra.run.smithi029.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'clients': ['ubuntu@smithi029.front.sepia.ceph.com'], 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'head': 'ubuntu@smithi029.front.sepia.ceph.com', 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'is_teuthology': True, 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'iterations': 1, 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'mons': ['ubuntu@smithi029.front.sepia.ceph.com'], 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'osds': ['ubuntu@smithi029.front.sepia.ceph.com'], 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'osds_per_node': 3, 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'pid_dir': '/var/run/ceph', 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'pgp_size': 256, 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'replication': 'replicated'}}, 2024-01-17T16:11:58.391 INFO:teuthology.orchestra.run.smithi029.stderr: 'rebuild_every_test': False, 2024-01-17T16:11:58.392 INFO:teuthology.orchestra.run.smithi029.stderr: 'rgws': {'ubuntu@smithi029.front.sepia.ceph.com': None}, 2024-01-17T16:11:58.392 INFO:teuthology.orchestra.run.smithi029.stderr: 'tmp_dir': '/tmp/cbt', 2024-01-17T16:11:58.392 INFO:teuthology.orchestra.run.smithi029.stderr: 'use_existing': True, 2024-01-17T16:11:58.392 INFO:teuthology.orchestra.run.smithi029.stderr: 'user': 'ubuntu'} 2024-01-17T16:11:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-01-17T16:11:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:11:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:11:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-01-17T16:11:58.415 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:11:58.415 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-17T16:11:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:11:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:11:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-01-17T16:11:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:11:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-17T16:11:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:16:11:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-17T16:12:58.481 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.482 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-17T16:12:58.525 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.526 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-17T16:12:58.565 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.565 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-01-17T16:12:58.566 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.567 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-01-17T16:12:58.569 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.569 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.569 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-01-17T16:12:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi029.front.sepia.ceph.com ; done 2024-01-17T16:12:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.590 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-01-17T16:12:58.624 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - WARNING - cbt - sudo killall -9 rados 2024-01-17T16:12:58.624 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - WARNING - cbt - error 1 seen, continuing anyway... 2024-01-17T16:12:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2024-01-17T16:12:58.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-01-17T16:12:58.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-01-17T16:12:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-01-17T16:12:58.662 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.662 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-17T16:12:58.664 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.664 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.664 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-01-17T16:12:58.667 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.668 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-17T16:12:58.668 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-17T16:12:58.668 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.668 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.668 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-01-17T16:12:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:12:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:16:12:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-01-17T16:13:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:00 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:00 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-01-17T16:13:01.105 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:01 - INFO - cbt - Checking Health after pool creation. 2024-01-17T16:13:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:06 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:06 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-17T16:13:06.433 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:06 - INFO - cbt - Final Pool Health Check. 2024-01-17T16:13:11.439 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.439 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.439 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-17T16:13:11.758 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.758 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-17T16:13:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-17T16:13:11.809 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.810 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-01-17T16:13:11.811 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.811 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-01-17T16:13:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:11.813 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-01-17T16:13:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-01-17T16:13:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:12 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:12 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:12.890 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-01-17T16:13:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:13 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:13 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:13.370 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-01-17T16:13:13.444 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:13 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:13.444 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:13 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:13.444 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-01-17T16:13:14.427 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.427 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.427 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-01-17T16:13:14.430 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.430 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-17T16:13:14.432 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.432 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.432 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - INFO - cbt - Running radosbench write test. 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.435 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-01-17T16:13:14.437 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.437 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:13:14.437 INFO:teuthology.orchestra.run.smithi029.stderr:16:13:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-01-17T16:14:15.573 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-17T16:14:15.605 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.605 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-17T16:14:15.641 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.641 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-01-17T16:14:15.642 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.642 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-01-17T16:14:15.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.644 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-01-17T16:14:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-01-17T16:14:15.972 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - get_fqdn_local()=smithi029.front.sepia.ceph.com 2024-01-17T16:14:15.972 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi029.front.sepia.ceph.com ; done 2024-01-17T16:14:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:16 - INFO - cbt - Convert results to json format. 2024-01-17T16:14:16.006 INFO:teuthology.orchestra.run.smithi029.stderr:16:14:16 - DEBUG - cbt - Nodes : ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:16.031 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-01-17T16:14:16.041 DEBUG:teuthology.orchestra.run.smithi029:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-01-17T16:14:16.269 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-01-17T16:14:16.269 INFO:tasks.cbt_performance:reading cbt preferences 2024-01-17T16:14:16.269 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-01-17T16:14:16.331 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi029.front.sepia.ceph.com 2024-01-17T16:14:16.331 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi029.front.sepia.ceph.com 2024-01-17T16:14:16.332 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:14:16.332 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi029.front.sepia.ceph.com of=/dev/stdout 2024-01-17T16:14:16.356 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-17T16:14:16.356 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2024-01-17T16:14:16.411 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-17T16:14:16.411 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-17T16:14:16.411 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-01-17T16:14:16.411 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi029.front.sepia.ceph.com of=/dev/stdout 2024-01-17T16:14:16.466 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-17T16:14:16.466 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2024-01-17T16:14:16.521 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-17T16:14:16.521 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-17T16:14:16.531 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-17T16:14:16.537 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-17T16:14:16.538 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-17T16:14:16.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-17T16:14:16.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:14:16.663+0000 7f2cf1315640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:14:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-01-17T16:14:16.671+0000 7f2cf1315640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-17T16:14:16.874 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-17T16:14:16.875 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-17T16:14:16.875 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-17T16:14:16.875 INFO:teuthology.orchestra.run:waiting for 300 2024-01-17T16:14:22.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-17T16:14:22.977 INFO:tasks.ceph.osd.0:Stopped 2024-01-17T16:14:22.977 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-17T16:14:22.977 INFO:teuthology.orchestra.run:waiting for 300 2024-01-17T16:14:27.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-17T16:14:27.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-17T16:14:29.079 INFO:tasks.ceph.osd.1:Stopped 2024-01-17T16:14:29.079 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-17T16:14:29.080 INFO:teuthology.orchestra.run:waiting for 300 2024-01-17T16:14:32.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-17T16:14:32.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-17T16:14:35.181 INFO:tasks.ceph.osd.2:Stopped 2024-01-17T16:14:35.181 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-17T16:14:35.182 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-17T16:14:35.182 INFO:teuthology.orchestra.run:waiting for 300 2024-01-17T16:14:35.213 INFO:tasks.ceph.mgr.x:Stopped 2024-01-17T16:14:35.213 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-17T16:14:35.214 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-17T16:14:35.214 INFO:teuthology.orchestra.run:waiting for 300 2024-01-17T16:14:35.266 INFO:tasks.ceph.mon.a:Stopped 2024-01-17T16:14:35.266 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-17T16:14:35.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-17T16:14:35.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:35.349 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-17T16:14:35.613 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:35.613 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-17T16:14:35.698 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:35.698 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-17T16:14:35.753 INFO:tasks.ceph:Archiving mon data... 2024-01-17T16:14:35.754 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868/data/mon.a.tgz 2024-01-17T16:14:35.755 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-01-17T16:14:35.781 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.lSV1eC9IOy 2024-01-17T16:14:35.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.lSV1eC9IOy 2024-01-17T16:14:35.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.lSV1eC9IOy 2024-01-17T16:14:36.018 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.lSV1eC9IOy is 175KB 2024-01-17T16:14:36.074 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.lSV1eC9IOy 2024-01-17T16:14:36.089 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-17T16:14:36.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-17T16:14:36.219 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-17T16:14:36.219 INFO:tasks.ceph:Archiving crash dumps... 2024-01-17T16:14:36.221 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868/remote/smithi029/crash 2024-01-17T16:14:36.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-17T16:14:36.286 INFO:tasks.ceph:Compressing logs... 2024-01-17T16:14:36.286 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-17T16:14:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-17T16:14:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27631.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29428.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.27631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28992.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31542.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.28992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30444.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29230.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.31542.log.gz 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29921.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30189.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29963.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29270.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30108.log 2024-01-17T16:14:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30066.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31431.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30315.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30024.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31394.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31478.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29190.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30190.log 2024-01-17T16:14:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30486.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31455.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31857.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29865.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29458.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30402.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-17T16:14:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-17T16:14:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-17T16:14:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-17T16:14:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30188.log 2024-01-17T16:14:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2024-01-17T16:14:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31329.log 2024-01-17T16:14:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31543.log 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29963.log: /var/log/ceph/ceph-client.admin.30189.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.30486.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.30402.log: /var/log/ceph/ceph-client.admin.31857.log: /var/log/ceph/ceph-client.admin.29080.log: /var/log/ceph/ceph-mgr.x.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29963.log.gz/var/log/ceph/ceph-osd.0.log: 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28992.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31543.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29428.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29921.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr: 67.2% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27631.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29230.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-17T16:14:36.409 INFO:teuthology.orchestra.run.smithi029.stderr: 65.3% -- replaced with /var/log/ceph/ceph-client.admin.31543.log.gz 2024-01-17T16:14:36.410 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30444.log.gz -- replaced with /var/log/ceph/ceph-client.admin.30189.log.gz 2024-01-17T16:14:36.410 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:14:36.410 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30276.log: /var/log/ceph/ceph-client.admin.31431.log: /var/log/ceph/ceph-client.admin.29270.log: /var/log/ceph/ceph-client.admin.30066.log: /var/log/ceph/ceph-client.admin.30108.log: /var/log/ceph/ceph-client.admin.30316.log: /var/log/ceph/ceph-client.admin.30024.log: 85.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-17T16:14:36.415 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31478.log: /var/log/ceph/ceph-client.admin.31394.log: /var/log/ceph/ceph-client.admin.30315.log: /var/log/ceph/ceph-client.admin.30190.log: /var/log/ceph/ceph-client.admin.29458.log: /var/log/ceph/ceph-client.admin.29190.log: /var/log/ceph/ceph-client.admin.29865.log: /var/log/ceph/ceph-client.admin.30188.log: /var/log/ceph/ceph-client.admin.31329.log: /var/log/ceph/ceph-client.admin.31455.log: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-17T16:14:36.425 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31478.log.gz 2024-01-17T16:14:36.436 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29190.log.gz 2024-01-17T16:14:36.448 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29270.log.gz 2024-01-17T16:14:36.452 INFO:teuthology.orchestra.run.smithi029.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29458.log.gz 2024-01-17T16:14:36.467 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31329.log.gz 2024-01-17T16:14:36.467 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31394.log.gz 2024-01-17T16:14:36.468 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz 2024-01-17T16:14:36.468 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31455.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29865.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30066.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30108.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30188.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30190.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30402.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30486.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gz 2024-01-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-01-17T16:14:36.489 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30315.log.gz 2024-01-17T16:14:36.489 INFO:teuthology.orchestra.run.smithi029.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30024.log.gz 2024-01-17T16:14:36.532 INFO:teuthology.orchestra.run.smithi029.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-17T16:14:37.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-17T16:14:38.151 INFO:teuthology.orchestra.run.smithi029.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-17T16:14:38.298 INFO:teuthology.orchestra.run.smithi029.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-17T16:14:38.375 INFO:teuthology.orchestra.run.smithi029.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-17T16:14:38.377 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-01-17T16:14:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m2.035s 2024-01-17T16:14:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m3.703s 2024-01-17T16:14:38.377 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.177s 2024-01-17T16:14:38.378 INFO:tasks.ceph:Archiving logs... 2024-01-17T16:14:38.378 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868/remote/smithi029/log 2024-01-17T16:14:38.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-17T16:14:39.010 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-17T16:14:39.019 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-17T16:14:39.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-17T16:14:39.073 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-17T16:14:39.081 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-17T16:14:39.081 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-17T16:14:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-01-17T16:14:39.177 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-17T16:14:39.178 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-01-17T16:14:39.178 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 -1465us[-1432us] +/- 90ms 2024-01-17T16:14:39.178 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 +242us[ +251us] +/- 51ms 2024-01-17T16:14:39.178 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +834us[ +866us] +/- 79ms 2024-01-17T16:14:39.178 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-17T16:14:39.204 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-17T16:14:39.213 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-17T16:14:39.214 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-17T16:14:39.237 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-17T16:14:39.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-17T16:14:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-17T16:14:39.423 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-17T16:14:39.432 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-17T16:14:39.454 INFO:teuthology.task.internal:Duration was 1034.101660 seconds 2024-01-17T16:14:39.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-17T16:14:39.487 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-17T16:14:39.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-17T16:14:39.621 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-17T16:14:39.937 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-17T16:14:39.937 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-01-17T16:14:39.938 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-17T16:14:39.957 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-17T16:14:39.957 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-17T16:14:40.025 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-17T16:14:40.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-17T16:14:40.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-17T16:14:40.175 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-17T16:14:40.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-17T16:14:40.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-17T16:14:40.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-17T16:14:40.291 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-01-17T16:14:40.322 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-17T16:14:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-17T16:14:40.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-17T16:14:40.387 INFO:teuthology.task.internal:Transferring archived files... 2024-01-17T16:14:40.387 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-01-17_15:47:21-crimson-rados-wip-matanb-crimson-testing-17.1-crimson-only-distro-crimson-smithi/7518868/remote/smithi029 2024-01-17T16:14:40.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-17T16:14:40.488 INFO:teuthology.task.internal:Removing archive directory... 2024-01-17T16:14:40.488 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-17T16:14:40.534 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-17T16:14:40.547 INFO:teuthology.task.internal:Not uploading archives. 2024-01-17T16:14:40.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-17T16:14:40.563 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-17T16:14:40.563 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-17T16:14:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 17 16:14 /home/ubuntu/cephtest 2024-01-17T16:14:40.589 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-17T16:14:40.600 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-17T16:14:40.666 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-testing-17.1-crimson-only&var-job_id_selected=7518868 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 1034.101660490036 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-01-17T16:14:40.667 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-17T16:14:40.738 INFO:teuthology.run:pass