2024-04-02T12:03:40.908 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T12:03:40.908 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_matan@teuthology --verbose --name matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} --archive /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514 -- /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514/orig.config.yaml 2024-04-02T12:03:40.953 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T12:03:41.050 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514 branch: wip-matanb-crimson-testing-apr-2 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} email: null first_in_suite: false job_id: '7635514' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-apr-2 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 107e1624f2a9b72a56ee529969558944d24a2a75 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 223 sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 107e1624f2a9b72a56ee529969558944d24a2a75 targets: smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEiGxHP0FnGWqmkNH1+sIt/K3egsTYNfXYVR2ZCqnR/vgXYWZ9TiBLbLLdTHORVMhIGS/ETnyL9HPJXo0ji0hyg= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_11:03:04 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T12:03:41.050 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa; will attempt to use it 2024-04-02T12:03:41.051 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa/tasks 2024-04-02T12:03:41.051 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T12:03:41.053 INFO:teuthology.task.internal:Checking packages... 2024-04-02T12:03:41.077 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '38caedd896c6b19ed5e43655372d95dc8e78cfda' 2024-04-02T12:03:41.077 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T12:03:41.077 INFO:teuthology.packaging:ref: None 2024-04-02T12:03:41.077 INFO:teuthology.packaging:tag: None 2024-04-02T12:03:41.077 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-apr-2 2024-04-02T12:03:41.077 INFO:teuthology.packaging:sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda 2024-04-02T12:03:41.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-apr-2 2024-04-02T12:03:41.298 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2589.g38caedd8 2024-04-02T12:03:41.300 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T12:03:41.311 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T12:03:41.311 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T12:03:41.330 INFO:teuthology.task.internal:Saving configuration 2024-04-02T12:03:41.349 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T12:03:41.355 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T12:03:41.381 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 11:58:33.154312', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEiGxHP0FnGWqmkNH1+sIt/K3egsTYNfXYVR2ZCqnR/vgXYWZ9TiBLbLLdTHORVMhIGS/ETnyL9HPJXo0ji0hyg='} 2024-04-02T12:03:41.381 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T12:03:41.389 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-02T12:03:41.389 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T12:03:41.439 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f128665cf70>, signals=[15]) 2024-04-02T12:03:41.439 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T12:03:41.446 INFO:teuthology.task.internal:Opening connections... 2024-04-02T12:03:41.446 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:03:41.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:03:41.549 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T12:03:41.556 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-04-02T12:03:41.619 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-04-02T12:03:41.619 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-04-02T12:03:41.676 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-04-02T12:03:41.676 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9" 2024-04-02T12:03:41.676 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-04-02T12:03:41.676 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-04-02T12:03:41.676 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T12:03:41.677 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T12:03:41.678 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-04-02T12:03:41.713 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T12:03:41.721 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T12:03:41.727 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T12:03:41.727 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-04-02T12:03:41.743 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T12:03:41.748 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T12:03:41.749 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-04-02T12:03:41.901 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T12:03:41.933 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T12:03:41.947 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-02T12:03:41.947 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-02T12:03:41.947 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T12:03:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:03:41.963 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-04-02T12:03:42.021 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-361.el9.x86_64 2024-04-02T12:03:42.021 INFO:teuthology.task.kernel:Running kernel on smithi159: 5.14.0-361.el9.x86_64 2024-04-02T12:03:42.021 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-04-02T12:03:44.036 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-02T12:03:45.695 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-04-02T12:03:46.569 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-02T12:03:49.353 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-04-02T12:04:00.928 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 776 kB/s | 6.1 MB 00:08 2024-04-02T12:06:12.962 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 122 B/s | 16 kB 02:10 2024-04-02T12:06:14.223 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-02T12:06:19.270 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-02T12:06:21.560 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T12:06:21.561 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T12:06:21.657 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:06:21.662 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout:Install 4 Packages 2024-04-02T12:06:21.663 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:06:21.664 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 76 M 2024-04-02T12:06:21.664 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 123 M 2024-04-02T12:06:21.664 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:06:21.821 INFO:teuthology.orchestra.run.smithi159.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-02T12:06:22.463 INFO:teuthology.orchestra.run.smithi159.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-04-02T12:06:23.247 INFO:teuthology.orchestra.run.smithi159.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-04-02T12:06:23.688 INFO:teuthology.orchestra.run.smithi159.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-04-02T12:06:23.691 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:06:23.691 INFO:teuthology.orchestra.run.smithi159.stdout:Total 38 MB/s | 76 MB 00:02 2024-04-02T12:06:23.692 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:06:24.086 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:06:24.086 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:06:24.917 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:06:24.917 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:06:27.204 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:06:27.869 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T12:06:27.948 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T12:06:30.761 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T12:06:30.908 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T12:06:35.617 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T12:06:35.657 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:06:40.398 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:07:09.584 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:07:09.584 INFO:teuthology.orchestra.run.smithi159.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T12:07:09.584 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:07:09.629 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:07:10.026 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:07:10.027 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T12:07:10.027 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T12:07:10.027 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T12:07:10.445 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:07:10.446 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:07:10.755 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T12:07:10.895 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T12:07:10.895 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T12:07:10.895 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T12:07:10.895 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T12:07:10.895 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T12:07:10.895 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-02T12:07:10.896 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T12:07:10.896 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T12:07:10.975 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T12:07:10.975 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T12:07:10.975 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.06 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 68.el9 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Group : Unspecified 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 8264254 2024-04-02T12:07:11.010 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : builder@centos.org 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-04-02T12:07:11.011 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:07:11.012 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T12:07:11.012 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T12:07:11.012 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 5.14.0-434.el9.x86_64 2024-04-02T12:07:11.012 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T12:07:11.630 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-04-02T12:07:12.999 INFO:teuthology.orchestra.run.smithi159.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T12:07:13.028 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-04-02T12:07:13.031 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-04-02T12:07:13.046 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.60RvLC7hqw 2024-04-02T12:07:13.046 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.60RvLC7hqw 2024-04-02T12:07:13.114 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.60RvLC7hqw 2024-04-02T12:07:13.250 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.60RvLC7hqw is 6KB 2024-04-02T12:07:13.299 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.60RvLC7hqw 2024-04-02T12:07:13.313 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T12:07:13.382 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T12:07:13.382 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T12:07:13.382 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T12:07:13.382 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T12:07:13.383 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T12:07:13.525 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-04-02T12:07:43.557 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-02T12:07:43.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:07:43.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:08:02.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-04-02T12:08:11.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:08:11.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:08:14.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-04-02T12:08:26.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:08:26.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:08:44.575 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-04-02T12:08:59.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:08:59.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:09:00.019 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-04-02T12:09:00.369 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:09:00.369 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-02T12:09:00.369 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-04-02T12:09:00.383 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-434.el9.x86_64 2024-04-02T12:09:00.384 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T12:09:00.384 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T12:09:00.384 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T12:09:01.384 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-02T12:09:01.384 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T12:09:01.457 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2024-04-02T12:09:01.508 DEBUG:teuthology.parallel:result is None 2024-04-02T12:09:01.508 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T12:09:01.516 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T12:09:01.517 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T12:09:01.532 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T12:09:01.539 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T12:09:01.545 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T12:09:01.545 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T12:09:01.591 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T12:09:01.597 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T12:09:01.598 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T12:09:01.661 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T12:09:01.674 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T12:09:01.676 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T12:09:01.683 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T12:09:01.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T12:09:01.741 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T12:09:01.751 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T12:09:01.751 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T12:09:01.793 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T12:09:01.876 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T12:09:01.920 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T12:09:02.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T12:09:02.040 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:09:02.040 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T12:09:02.107 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-04-02T12:09:02.195 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T12:09:02.428 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T12:09:02.438 INFO:teuthology.task.internal:Starting timer... 2024-04-02T12:09:02.438 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T12:09:02.449 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T12:09:02.457 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-02T12:09:02.458 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-04-02T12:09:02.536 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-04-02T12:09:02.538 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T12:09:02.539 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-04-02T12:09:02.558 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-04-02T12:09:02.559 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-04-02T12:09:02.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T12:09:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:09:02.623 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T12:09:02.623 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-04-02T12:09:02.718 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T12:09:02.728 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T12:09:02.745 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T12:09:02.766 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T12:13:13.946 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-04-02T12:13:13.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:13:13.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:13:14.041 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-04-02T12:13:14.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-04-02T12:13:14.118 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T12:13:14.128 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T12:13:14.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T12:13:14.129 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T12:13:14.188 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T12:13:14.204 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T12:13:14.234 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-04-02T12:13:14.248 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-04-02T12:13:14.265 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T12:13:14.281 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T12:13:14.335 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-04-02T12:13:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T12:13:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-04-02T12:13:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:13:14.338 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:13:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:13:14.339 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:13:14.340 INFO:teuthology.run_tasks:Running task install... 2024-04-02T12:13:14.348 DEBUG:teuthology.task.install:project ceph 2024-04-02T12:13:14.348 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'}} 2024-04-02T12:13:14.348 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'} 2024-04-02T12:13:14.348 INFO:teuthology.task.install:Using flavor: crimson 2024-04-02T12:13:14.354 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T12:13:14.354 INFO:teuthology.task.install:extra packages: [] 2024-04-02T12:13:14.354 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda', 'tag': None, 'wait_for_package': False} 2024-04-02T12:13:14.355 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=38caedd896c6b19ed5e43655372d95dc8e78cfda 2024-04-02T12:13:14.502 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/ 2024-04-02T12:13:14.502 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2589.g38caedd8 2024-04-02T12:13:14.754 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T12:13:14.755 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:13:14.755 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T12:13:14.782 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-02T12:13:14.783 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38caedd896c6b19ed5e43655372d95dc8e78cfda/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T12:13:14.851 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T12:13:14.932 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T12:13:14.998 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-04-02T12:13:15.000 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-04-02T12:13:15.250 INFO:teuthology.orchestra.run.smithi159.stdout:74 files removed 2024-04-02T12:13:15.280 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw 2024-04-02T12:13:15.914 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 260 kB/s | 87 kB 00:00 2024-04-02T12:13:16.531 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 20 kB/s | 12 kB 00:00 2024-04-02T12:13:16.865 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 6.9 kB/s | 2.0 kB 00:00 2024-04-02T12:13:17.450 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T12:13:19.043 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-04-02T12:13:19.817 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-04-02T12:13:21.618 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-02T12:13:26.396 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-04-02T12:13:27.838 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-02T12:13:28.856 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-02T12:13:33.806 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-02T12:13:36.434 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 9.6 M 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 3.4 M 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 3.7 M 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:13:36.445 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 5.9 M 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 19 M 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 25 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 795 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T12:13:36.446 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 548 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 5.0 M 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 45 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 134 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 159 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 315 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 294 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 99 k 2024-04-02T12:13:36.447 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout:Install 30 Packages 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 2 Packages 2024-04-02T12:13:36.448 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:13:36.449 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 58 M 2024-04-02T12:13:36.449 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:13:37.840 INFO:teuthology.orchestra.run.smithi159.stdout:(1/32): ceph-radosgw-19.0.0-2589.g38caedd8.el9. 7.7 MB/s | 9.6 MB 00:01 2024-04-02T12:13:38.050 INFO:teuthology.orchestra.run.smithi159.stdout:(2/32): ceph-common-19.0.0-2589.g38caedd8.el9.x 13 MB/s | 19 MB 00:01 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(3/32): ceph-selinux-19.0.0-2589.g38caedd8.el9. 107 kB/s | 25 kB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(4/32): libcephfs2-19.0.0-2589.g38caedd8.el9.x8 10 MB/s | 795 kB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(5/32): libradosstriper1-19.0.0-2589.g38caedd8. 7.1 MB/s | 548 kB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(6/32): python3-ceph-argparse-19.0.0-2589.g38ca 676 kB/s | 45 kB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(7/32): python3-ceph-common-19.0.0-2589.g38caed 1.1 MB/s | 134 kB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(8/32): librgw2-19.0.0-2589.g38caedd8.el9.x86_6 17 MB/s | 5.0 MB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(9/32): python3-cephfs-19.0.0-2589.g38caedd8.el 1.6 MB/s | 159 kB 00:00 2024-04-02T12:13:38.868 INFO:teuthology.orchestra.run.smithi159.stdout:(10/32): python3-rados-19.0.0-2589.g38caedd8.el 6.1 MB/s | 315 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(11/32): python3-rbd-19.0.0-2589.g38caedd8.el9. 4.3 MB/s | 294 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(12/32): python3-rgw-19.0.0-2589.g38caedd8.el9. 1.7 MB/s | 99 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 831 kB/s | 41 kB 00:00 2024-04-02T12:13:38.869 INFO:teuthology.orchestra.run.smithi159.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-02T12:13:38.888 INFO:teuthology.orchestra.run.smithi159.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-04-02T12:13:38.980 INFO:teuthology.orchestra.run.smithi159.stdout:(21/32): ceph-base-19.0.0-2589.g38caedd8.el9.x8 2.5 MB/s | 5.9 MB 00:02 2024-04-02T12:13:39.005 INFO:teuthology.orchestra.run.smithi159.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 326 kB/s | 46 kB 00:00 2024-04-02T12:13:39.430 INFO:teuthology.orchestra.run.smithi159.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 61 kB/s | 26 kB 00:00 2024-04-02T12:13:39.613 INFO:teuthology.orchestra.run.smithi159.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-04-02T12:13:39.772 INFO:teuthology.orchestra.run.smithi159.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 426 kB/s | 67 kB 00:00 2024-04-02T12:13:39.806 INFO:teuthology.orchestra.run.smithi159.stdout:(26/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 336 kB/s | 309 kB 00:00 2024-04-02T12:13:40.048 INFO:teuthology.orchestra.run.smithi159.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 793 kB/s | 192 kB 00:00 2024-04-02T12:13:40.073 INFO:teuthology.orchestra.run.smithi159.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-04-02T12:13:40.140 INFO:teuthology.orchestra.run.smithi159.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-04-02T12:13:40.415 INFO:teuthology.orchestra.run.smithi159.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2024-04-02T12:13:40.482 INFO:teuthology.orchestra.run.smithi159.stdout:(31/32): librbd1-19.0.0-2589.g38caedd8.el9.x86_ 11 MB/s | 3.7 MB 00:00 2024-04-02T12:13:40.716 INFO:teuthology.orchestra.run.smithi159.stdout:(32/32): librados2-19.0.0-2589.g38caedd8.el9.x8 5.3 MB/s | 3.4 MB 00:00 2024-04-02T12:13:40.723 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:13:40.723 INFO:teuthology.orchestra.run.smithi159.stdout:Total 14 MB/s | 58 MB 00:04 2024-04-02T12:13:40.724 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:13:40.762 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:13:40.762 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:13:41.260 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:13:41.260 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:13:42.728 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:13:43.233 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-02T12:13:43.276 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 2/34 2024-04-02T12:13:43.351 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 2/34 2024-04-02T12:13:43.439 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 3/34 2024-04-02T12:13:43.469 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 4/34 2024-04-02T12:13:43.533 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 4/34 2024-04-02T12:13:43.574 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-02T12:13:43.609 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-02T12:13:43.646 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-02T12:13:43.664 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-02T12:13:43.680 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-02T12:13:43.831 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-02T12:13:43.863 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-02T12:13:44.235 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-02T12:13:44.268 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 11/34 2024-04-02T12:13:44.335 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 11/34 2024-04-02T12:13:44.378 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:19.0.0-2589.g38caedd8.el9. 12/34 2024-04-02T12:13:44.433 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 13/34 2024-04-02T12:13:44.465 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 14/34 2024-04-02T12:13:44.545 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-02T12:13:44.570 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_6 16/34 2024-04-02T12:13:44.659 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_6 16/34 2024-04-02T12:13:44.694 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-02T12:13:44.803 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-02T12:13:44.837 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-02T12:13:44.869 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-02T12:13:44.901 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-02T12:13:44.930 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-02T12:13:44.966 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-02T12:13:45.011 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-02T12:13:45.544 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-02T12:13:45.569 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 26/34 2024-04-02T12:13:45.639 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 26/34 2024-04-02T12:13:45.708 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 27/34 2024-04-02T12:13:45.737 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:19.0.0-2589.g38caedd8.el9.x8 28/34 2024-04-02T12:13:48.088 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 29/34 2024-04-02T12:13:48.134 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 29/34 2024-04-02T12:13:48.827 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 29/34 2024-04-02T12:13:48.863 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 30/34 2024-04-02T12:13:48.955 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 30/34 2024-04-02T12:13:48.955 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T12:13:48.955 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T12:13:48.955 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:13:48.997 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 31/34 2024-04-02T12:13:58.900 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 31/34 2024-04-02T12:13:58.900 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-04-02T12:13:58.900 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-04-02T12:13:58.900 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-04-02T12:13:58.901 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-04-02T12:13:58.901 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-04-02T12:13:58.901 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /root 2024-04-02T12:13:58.901 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-04-02T12:13:58.901 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:13:59.921 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 32/34 2024-04-02T12:13:59.946 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 32/34 2024-04-02T12:13:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:13:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T12:13:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T12:13:59.946 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T12:13:59.946 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:13:59.949 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-02T12:14:00.024 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-02T12:14:00.024 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 1/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 2/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 3/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 4/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 5/34 2024-04-02T12:14:01.260 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_6 6/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 7/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2589.g38caedd8.el9. 8/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:19.0.0-2589.g38caedd8.el9.x8 9/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 10/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 11/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 12/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 13/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-02T12:14:01.261 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-02T12:14:01.262 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-02T12:14:01.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-02T12:14:01.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 31/34 2024-04-02T12:14:01.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-02T12:14:01.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 33/34 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T12:14:02.209 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.210 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T12:14:02.211 INFO:teuthology.orchestra.run.smithi159.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T12:14:02.212 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T12:14:02.212 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:02.212 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:14:02.541 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test 2024-04-02T12:14:03.106 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:30 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:14:03.259 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:14:03.260 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:14:03.260 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:14:03.260 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:14:03.260 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:14:03.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 87 M 2024-04-02T12:14:03.260 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 173 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout:Install 11 Packages 2024-04-02T12:14:03.261 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:03.262 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 90 M 2024-04-02T12:14:03.262 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 820 M 2024-04-02T12:14:03.262 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:14:03.746 INFO:teuthology.orchestra.run.smithi159.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-04-02T12:14:03.838 INFO:teuthology.orchestra.run.smithi159.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-02T12:14:03.880 INFO:teuthology.orchestra.run.smithi159.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-02T12:14:03.914 INFO:teuthology.orchestra.run.smithi159.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-02T12:14:03.955 INFO:teuthology.orchestra.run.smithi159.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.1 MB/s | 90 kB 00:00 2024-04-02T12:14:03.981 INFO:teuthology.orchestra.run.smithi159.stdout:(6/11): libcephsqlite-19.0.0-2589.g38caedd8.el9 386 kB/s | 173 kB 00:00 2024-04-02T12:14:04.206 INFO:teuthology.orchestra.run.smithi159.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.0 MB/s | 469 kB 00:00 2024-04-02T12:14:04.231 INFO:teuthology.orchestra.run.smithi159.stdout:(8/11): jq-1.6-16.el9.x86_64.rpm 679 kB/s | 187 kB 00:00 2024-04-02T12:14:04.323 INFO:teuthology.orchestra.run.smithi159.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-02T12:14:04.432 INFO:teuthology.orchestra.run.smithi159.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 608 kB/s | 122 kB 00:00 2024-04-02T12:14:09.182 INFO:teuthology.orchestra.run.smithi159.stdout:(11/11): ceph-test-19.0.0-2589.g38caedd8.el9.x8 15 MB/s | 87 MB 00:05 2024-04-02T12:14:09.184 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:14:09.185 INFO:teuthology.orchestra.run.smithi159.stdout:Total 15 MB/s | 90 MB 00:05 2024-04-02T12:14:09.228 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:14:09.283 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:14:09.283 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:14:09.885 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:14:09.885 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:14:10.933 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:14:11.001 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-02T12:14:11.036 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-02T12:14:11.058 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-02T12:14:11.141 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-02T12:14:11.175 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-02T12:14:11.219 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-02T12:14:11.264 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-02T12:14:11.303 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-02T12:14:11.343 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-02T12:14:11.379 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-02T12:14:11.405 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 10/11 2024-04-02T12:14:22.924 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 10/11 2024-04-02T12:14:23.003 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 11/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 11/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 1/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 2/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-02T12:14:29.314 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-02T12:14:29.315 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-02T12:14:29.315 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-02T12:14:29.315 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-02T12:14:29.688 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-02T12:14:29.688 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:29.688 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:14:29.688 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-16.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:29.689 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:14:29.991 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph 2024-04-02T12:14:30.558 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:57 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:14:30.711 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 6.5 k 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:14:30.713 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 2.2 M 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 1.6 M 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 249 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 5.0 M 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 71 M 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T12:14:30.714 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T12:14:30.715 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout:Install 36 Packages 2024-04-02T12:14:30.716 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:14:30.717 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 84 M 2024-04-02T12:14:30.717 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 922 M 2024-04-02T12:14:30.717 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:16:42.539 INFO:teuthology.orchestra.run.smithi159.stdout:(1/36): ceph-19.0.0-2589.g38caedd8.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2024-04-02T12:16:42.646 INFO:teuthology.orchestra.run.smithi159.stdout:(2/36): ceph-mgr-19.0.0-2589.g38caedd8.el9.x86_ 3.9 MB/s | 1.6 MB 00:00 2024-04-02T12:16:42.955 INFO:teuthology.orchestra.run.smithi159.stdout:(3/36): ceph-mds-19.0.0-2589.g38caedd8.el9.x86_ 3.1 MB/s | 2.2 MB 00:00 2024-04-02T12:16:43.038 INFO:teuthology.orchestra.run.smithi159.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2589.g38ca 2.9 MB/s | 249 kB 00:00 2024-04-02T12:16:43.172 INFO:teuthology.orchestra.run.smithi159.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-04-02T12:16:43.214 INFO:teuthology.orchestra.run.smithi159.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-02T12:16:43.248 INFO:teuthology.orchestra.run.smithi159.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-04-02T12:16:43.290 INFO:teuthology.orchestra.run.smithi159.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-02T12:16:43.332 INFO:teuthology.orchestra.run.smithi159.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2024-04-02T12:16:43.382 INFO:teuthology.orchestra.run.smithi159.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-02T12:16:43.499 INFO:teuthology.orchestra.run.smithi159.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-04-02T12:16:43.549 INFO:teuthology.orchestra.run.smithi159.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-04-02T12:16:43.600 INFO:teuthology.orchestra.run.smithi159.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-02T12:16:43.650 INFO:teuthology.orchestra.run.smithi159.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-02T12:16:43.809 INFO:teuthology.orchestra.run.smithi159.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 789 kB/s | 125 kB 00:00 2024-04-02T12:16:43.851 INFO:teuthology.orchestra.run.smithi159.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 538 kB/s | 22 kB 00:00 2024-04-02T12:16:44.118 INFO:teuthology.orchestra.run.smithi159.stdout:(17/36): ceph-mon-19.0.0-2589.g38caedd8.el9.x86 3.2 MB/s | 5.0 MB 00:01 2024-04-02T12:16:44.310 INFO:teuthology.orchestra.run.smithi159.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 329 kB/s | 151 kB 00:00 2024-04-02T12:16:44.418 INFO:teuthology.orchestra.run.smithi159.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-04-02T12:16:44.477 INFO:teuthology.orchestra.run.smithi159.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-04-02T12:16:44.577 INFO:teuthology.orchestra.run.smithi159.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-02T12:16:44.722 INFO:teuthology.orchestra.run.smithi159.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 122 kB/s | 18 kB 00:00 2024-04-02T12:16:44.786 INFO:teuthology.orchestra.run.smithi159.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 976 kB/s | 358 kB 00:00 2024-04-02T12:16:44.819 INFO:teuthology.orchestra.run.smithi159.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 238 kB/s | 23 kB 00:00 2024-04-02T12:16:44.886 INFO:teuthology.orchestra.run.smithi159.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-04-02T12:16:44.928 INFO:teuthology.orchestra.run.smithi159.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-04-02T12:16:45.004 INFO:teuthology.orchestra.run.smithi159.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-04-02T12:16:45.071 INFO:teuthology.orchestra.run.smithi159.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 556 kB/s | 79 kB 00:00 2024-04-02T12:16:45.204 INFO:teuthology.orchestra.run.smithi159.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-04-02T12:16:45.246 INFO:teuthology.orchestra.run.smithi159.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-04-02T12:16:45.346 INFO:teuthology.orchestra.run.smithi159.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 634 kB/s | 90 kB 00:00 2024-04-02T12:16:45.380 INFO:teuthology.orchestra.run.smithi159.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 268 kB/s | 36 kB 00:00 2024-04-02T12:16:46.047 INFO:teuthology.orchestra.run.smithi159.stdout:(33/36): ceph-osd-19.0.0-2589.g38caedd8.el9.x86 21 MB/s | 71 MB 00:03 2024-04-02T12:16:46.072 INFO:teuthology.orchestra.run.smithi159.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 316 kB/s | 230 kB 00:00 2024-04-02T12:16:46.097 INFO:teuthology.orchestra.run.smithi159.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 596 kB/s | 427 kB 00:00 2024-04-02T12:16:46.373 INFO:teuthology.orchestra.run.smithi159.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 61 kB/s | 20 kB 00:00 2024-04-02T12:16:46.377 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:16:46.377 INFO:teuthology.orchestra.run.smithi159.stdout:Total 636 kB/s | 84 MB 02:15 2024-04-02T12:16:46.398 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:16:46.442 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:16:46.442 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:16:47.175 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:16:47.175 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:16:48.826 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:16:48.892 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-02T12:16:48.936 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-02T12:16:48.969 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-02T12:16:49.042 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-02T12:16:49.110 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-02T12:16:49.143 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-02T12:16:49.234 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-02T12:16:49.313 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-02T12:16:49.354 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-02T12:16:49.426 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-02T12:16:49.488 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-02T12:16:49.526 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-02T12:16:49.583 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-02T12:16:49.639 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-02T12:16:49.827 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-02T12:16:49.881 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-02T12:16:49.925 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-02T12:16:49.962 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-02T12:16:49.992 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-02T12:16:50.069 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-02T12:16:50.102 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-02T12:16:50.135 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-02T12:16:50.175 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-02T12:16:50.211 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-02T12:16:50.254 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-02T12:16:50.344 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-02T12:16:50.385 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-02T12:16:50.457 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-02T12:16:50.593 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-02T12:16:50.719 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-02T12:16:50.905 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2589.g38caedd8.el9. 31/36 2024-04-02T12:16:50.931 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 32/36 2024-04-02T12:16:50.956 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 32/36 2024-04-02T12:16:50.956 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:16:50.956 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T12:16:50.956 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T12:16:50.956 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T12:16:50.956 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:01.890 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 33/36 2024-04-02T12:17:01.917 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 33/36 2024-04-02T12:17:01.917 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:17:01.917 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T12:17:01.918 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T12:17:01.918 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T12:17:01.918 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:02.449 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 34/36 2024-04-02T12:17:02.472 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 34/36 2024-04-02T12:17:02.472 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:17:02.472 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T12:17:02.472 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T12:17:02.472 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T12:17:02.473 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:02.708 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 35/36 2024-04-02T12:17:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 35/36 2024-04-02T12:17:02.731 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:17:02.731 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T12:17:02.731 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T12:17:02.731 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T12:17:02.731 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:02.798 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 36/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 36/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 1/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 2/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 3/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 4/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 5/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2589.g38caedd8.el9. 6/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-02T12:17:09.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-02T12:17:09.652 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-02T12:17:09.653 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-02T12:17:09.653 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-02T12:17:09.653 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-02T12:17:09.653 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-02T12:17:09.654 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:10.680 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T12:17:10.681 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T12:17:10.682 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T12:17:10.683 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T12:17:10.683 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T12:17:10.683 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T12:17:10.683 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T12:17:10.683 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:10.683 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:10.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base 2024-04-02T12:17:11.493 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:38 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:11.591 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:17:11.644 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:11.646 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:17:11.646 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:11.717 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install cephadm 2024-04-02T12:17:12.277 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:39 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:12.426 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 762 k 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:12.427 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:17:12.428 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:12.428 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 762 k 2024-04-02T12:17:12.428 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 768 k 2024-04-02T12:17:12.428 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:12.851 INFO:teuthology.orchestra.run.smithi159.stdout:cephadm-19.0.0-2589.g38caedd8.el9.noarch.rpm 1.8 MB/s | 762 kB 00:00 2024-04-02T12:17:12.852 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:12.852 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.8 MB/s | 762 kB 00:00 2024-04-02T12:17:12.852 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:12.857 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:12.857 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:12.863 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:12.863 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:12.933 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:13.427 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:17:13.505 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:17:13.907 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:17:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:17:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:14.190 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:14.190 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:14.190 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:14.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache 2024-04-02T12:17:14.965 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:41 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:15.119 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 157 k 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:17:15.120 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 157 k 2024-04-02T12:17:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 640 k 2024-04-02T12:17:15.121 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:15.349 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-19.0.0-2589.g38caed 690 kB/s | 157 kB 00:00 2024-04-02T12:17:15.350 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:15.350 INFO:teuthology.orchestra.run.smithi159.stdout:Total 685 kB/s | 157 kB 00:00 2024-04-02T12:17:15.350 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:15.359 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:15.360 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:15.411 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:15.411 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:15.550 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:15.600 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8. 1/1 2024-04-02T12:17:15.625 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8. 1/1 2024-04-02T12:17:15.625 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:17:15.625 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T12:17:15.625 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8. 1/1 2024-04-02T12:17:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:16.588 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:16.588 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:16.589 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:16.589 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:16.826 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr 2024-04-02T12:17:17.401 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:44 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:17.502 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:17:17.554 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:17.555 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:17:17.555 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:17.637 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-dashboard 2024-04-02T12:17:18.202 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:45 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:18.355 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:18.356 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-04-02T12:17:18.356 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:18.356 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 3.5 M 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 34 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 16 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T12:17:18.357 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout:Install 11 Packages 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 7.6 M 2024-04-02T12:17:18.358 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 100 M 2024-04-02T12:17:18.359 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:18.887 INFO:teuthology.orchestra.run.smithi159.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2589.g38c 43 kB/s | 16 kB 00:00 2024-04-02T12:17:18.987 INFO:teuthology.orchestra.run.smithi159.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2589.g38 70 kB/s | 34 kB 00:00 2024-04-02T12:17:19.012 INFO:teuthology.orchestra.run.smithi159.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-04-02T12:17:19.204 INFO:teuthology.orchestra.run.smithi159.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-04-02T12:17:19.580 INFO:teuthology.orchestra.run.smithi159.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-04-02T12:17:19.755 INFO:teuthology.orchestra.run.smithi159.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 738 kB/s | 548 kB 00:00 2024-04-02T12:17:19.872 INFO:teuthology.orchestra.run.smithi159.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-04-02T12:17:19.989 INFO:teuthology.orchestra.run.smithi159.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 263 kB/s | 31 kB 00:00 2024-04-02T12:17:20.106 INFO:teuthology.orchestra.run.smithi159.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-04-02T12:17:20.264 INFO:teuthology.orchestra.run.smithi159.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.0 MB/s | 2.0 MB 00:00 2024-04-02T12:17:20.831 INFO:teuthology.orchestra.run.smithi159.stdout:(11/11): ceph-mgr-dashboard-19.0.0-2589.g38caed 1.5 MB/s | 3.5 MB 00:02 2024-04-02T12:17:20.836 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:20.836 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.1 MB/s | 7.6 MB 00:02 2024-04-02T12:17:20.855 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:20.872 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:20.873 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:21.050 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:21.050 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:21.552 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:21.637 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-02T12:17:21.668 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-02T12:17:21.762 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-02T12:17:21.893 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-02T12:17:21.945 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-02T12:17:22.029 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-02T12:17:22.072 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-02T12:17:22.102 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-02T12:17:22.134 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2589.g38caedd8.el9 9/11 2024-04-02T12:17:23.174 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2589.g38caedd8.el 10/11 2024-04-02T12:17:23.217 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noa 11/11 2024-04-02T12:17:24.645 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noa 11/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2589.g38caedd8.el 1/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noa 2/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2589.g38caedd8.el9 3/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-02T12:17:24.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-02T12:17:24.647 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-02T12:17:24.981 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-02T12:17:24.981 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T12:17:24.982 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T12:17:24.983 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T12:17:24.983 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:24.983 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:25.230 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-02T12:17:25.797 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:52 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:25.952 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:25.953 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-04-02T12:17:25.953 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:25.953 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 7.4 M 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T12:17:25.954 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:25.955 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 37 M 2024-04-02T12:17:25.956 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 209 M 2024-04-02T12:17:25.956 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:26.430 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 218 kB/s | 32 kB 00:00 2024-04-02T12:17:26.480 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 348 kB/s | 17 kB 00:00 2024-04-02T12:17:26.531 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-04-02T12:17:26.723 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.8 MB/s | 3.0 MB 00:00 2024-04-02T12:17:26.848 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-04-02T12:17:26.940 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-04-02T12:17:26.965 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-04-02T12:17:27.174 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-25 8.3 MB/s | 7.4 MB 00:00 2024-04-02T12:17:27.374 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-04-02T12:17:27.750 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-02T12:17:27.791 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 318 kB/s | 195 kB 00:00 2024-04-02T12:17:28.266 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 275 kB/s | 245 kB 00:00 2024-04-02T12:17:28.270 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:28.270 INFO:teuthology.orchestra.run.smithi159.stdout:Total 16 MB/s | 37 MB 00:02 2024-04-02T12:17:28.315 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:28.340 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:28.340 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:28.694 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:28.694 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:29.392 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:29.456 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-02T12:17:29.496 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-02T12:17:29.589 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-02T12:17:29.633 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-02T12:17:29.814 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-02T12:17:29.840 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-02T12:17:29.924 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-02T12:17:30.564 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-02T12:17:30.671 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-02T12:17:31.919 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-02T12:17:32.942 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-02T12:17:32.973 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2589.g38cae 12/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2589.g38cae 12/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2589.g38cae 1/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-02T12:17:35.554 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-02T12:17:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-02T12:17:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-02T12:17:35.555 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T12:17:36.017 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T12:17:36.018 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T12:17:36.018 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T12:17:36.018 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T12:17:36.018 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:36.018 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:36.425 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-rook 2024-04-02T12:17:36.995 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:04 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:37.146 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 49 k 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T12:17:37.148 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-04-02T12:17:37.149 INFO:teuthology.orchestra.run.smithi159.stdout:Install 13 Packages 2024-04-02T12:17:37.150 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:37.150 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 2.0 M 2024-04-02T12:17:37.150 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2024-04-02T12:17:37.150 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:37.473 INFO:teuthology.orchestra.run.smithi159.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 18 kB 00:00 2024-04-02T12:17:37.498 INFO:teuthology.orchestra.run.smithi159.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 160 kB/s | 24 kB 00:00 2024-04-02T12:17:37.548 INFO:teuthology.orchestra.run.smithi159.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-02T12:17:37.573 INFO:teuthology.orchestra.run.smithi159.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-02T12:17:37.599 INFO:teuthology.orchestra.run.smithi159.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-02T12:17:37.624 INFO:teuthology.orchestra.run.smithi159.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-04-02T12:17:37.649 INFO:teuthology.orchestra.run.smithi159.stdout:(7/13): ceph-mgr-rook-19.0.0-2589.g38caedd8.el9 162 kB/s | 49 kB 00:00 2024-04-02T12:17:38.033 INFO:teuthology.orchestra.run.smithi159.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2024-04-02T12:17:38.058 INFO:teuthology.orchestra.run.smithi159.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2024-04-02T12:17:38.292 INFO:teuthology.orchestra.run.smithi159.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-04-02T12:17:38.317 INFO:teuthology.orchestra.run.smithi159.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 304 kB/s | 203 kB 00:00 2024-04-02T12:17:38.492 INFO:teuthology.orchestra.run.smithi159.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2024-04-02T12:17:38.684 INFO:teuthology.orchestra.run.smithi159.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-04-02T12:17:38.689 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:38.689 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-04-02T12:17:38.690 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:38.705 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:38.705 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:38.806 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:38.806 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:39.321 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:39.489 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-02T12:17:39.552 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-02T12:17:39.607 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-02T12:17:39.644 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-02T12:17:39.686 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-02T12:17:39.787 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-02T12:17:39.888 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-02T12:17:39.937 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-02T12:17:40.657 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-02T12:17:40.718 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-02T12:17:40.751 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-02T12:17:40.800 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-02T12:17:40.821 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 13/13 2024-04-02T12:17:41.618 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 13/13 2024-04-02T12:17:41.618 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 1/13 2024-04-02T12:17:41.618 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-02T12:17:41.618 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-02T12:17:41.618 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-02T12:17:41.618 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-02T12:17:41.619 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T12:17:41.971 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T12:17:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T12:17:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T12:17:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T12:17:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T12:17:41.972 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:41.972 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:42.210 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-cephadm 2024-04-02T12:17:42.778 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:09 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:42.934 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:42.934 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-04-02T12:17:42.934 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 141 k 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:42.935 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-04-02T12:17:42.936 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2024-04-02T12:17:42.936 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:42.936 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 6.8 M 2024-04-02T12:17:42.936 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 33 M 2024-04-02T12:17:42.936 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:43.310 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-02T12:17:43.511 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-04-02T12:17:43.569 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2589.g38caedd8.e 330 kB/s | 141 kB 00:00 2024-04-02T12:17:43.828 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-04-02T12:17:43.853 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-04-02T12:17:43.878 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-04-02T12:17:43.880 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:43.880 INFO:teuthology.orchestra.run.smithi159.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-04-02T12:17:43.893 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:43.903 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:43.903 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:43.996 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:43.997 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:44.289 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:44.444 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-02T12:17:44.521 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-02T12:17:45.037 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-02T12:17:45.124 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-02T12:17:45.187 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-02T12:17:45.217 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 6/6 2024-04-02T12:17:46.010 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 6/6 2024-04-02T12:17:46.010 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/6 2024-04-02T12:17:46.010 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-02T12:17:46.010 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-02T12:17:46.010 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-02T12:17:46.010 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T12:17:46.310 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T12:17:46.311 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T12:17:46.311 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:46.311 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:46.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse 2024-04-02T12:17:47.125 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:14 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:47.279 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:47.279 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:47.279 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:47.279 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:47.279 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:47.279 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 871 k 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:47.280 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 950 k 2024-04-02T12:17:47.281 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 4.0 M 2024-04-02T12:17:47.281 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:47.397 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 677 kB/s | 79 kB 00:00 2024-04-02T12:17:47.597 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-fuse-19.0.0-2589.g38caedd8.el9.x86_ 2.7 MB/s | 871 kB 00:00 2024-04-02T12:17:47.598 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:47.599 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.9 MB/s | 950 kB 00:00 2024-04-02T12:17:47.599 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:47.621 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:47.621 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:47.694 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:47.695 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:47.886 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:48.040 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-02T12:17:48.114 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 2/2 2024-04-02T12:17:48.771 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 2/2 2024-04-02T12:17:48.771 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/2 2024-04-02T12:17:49.069 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-02T12:17:49.070 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:49.070 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:49.070 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-02T12:17:49.070 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:49.070 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:49.334 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-volume 2024-04-02T12:17:49.915 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:16 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:50.073 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 264 k 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:50.074 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:50.075 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:50.075 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-04-02T12:17:50.075 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:50.075 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 332 k 2024-04-02T12:17:50.075 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.5 M 2024-04-02T12:17:50.075 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:50.252 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 391 kB/s | 69 kB 00:00 2024-04-02T12:17:50.352 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-volume-19.0.0-2589.g38caedd8.el9.no 955 kB/s | 264 kB 00:00 2024-04-02T12:17:50.353 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:50.353 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-04-02T12:17:50.353 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:50.362 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:50.362 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:50.402 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:50.403 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:50.625 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:50.785 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-02T12:17:50.821 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 2/2 2024-04-02T12:17:50.838 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 2/2 2024-04-02T12:17:50.838 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:17:50.838 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T12:17:50.838 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:51.579 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 1/2 2024-04-02T12:17:51.961 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-02T12:17:51.961 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:51.961 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:51.961 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:17:51.961 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T12:17:51.962 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:51.962 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:52.274 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel 2024-04-02T12:17:52.848 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:19 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:53.000 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:53.000 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 125 k 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 125 k 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 445 k 2024-04-02T12:17:53.001 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:53.210 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-19.0.0-2589.g38caedd8.el9.x86_64 598 kB/s | 125 kB 00:00 2024-04-02T12:17:53.211 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:53.211 INFO:teuthology.orchestra.run.smithi159.stdout:Total 594 kB/s | 125 kB 00:00 2024-04-02T12:17:53.211 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:53.217 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:53.217 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:53.277 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:53.277 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:53.426 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:53.537 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:17:54.040 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:17:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:17:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:54.335 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:54.335 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:54.336 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:54.336 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:54.554 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2 2024-04-02T12:17:55.133 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:22 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:55.235 INFO:teuthology.orchestra.run.smithi159.stdout:Package libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:17:55.287 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:55.288 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:17:55.288 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:55.357 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs-devel 2024-04-02T12:17:55.916 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:22 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:56.065 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:56.067 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:56.067 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-04-02T12:17:56.067 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:56.068 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:17:56.068 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 32 k 2024-04-02T12:17:56.068 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:56.068 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:17:56.068 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:17:56.068 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:17:56.069 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:56.069 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 32 k 2024-04-02T12:17:56.069 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 142 k 2024-04-02T12:17:56.069 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:17:56.231 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs-devel-19.0.0-2589.g38caedd8.el9.x86_6 194 kB/s | 32 kB 00:00 2024-04-02T12:17:56.231 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:56.232 INFO:teuthology.orchestra.run.smithi159.stdout:Total 192 kB/s | 32 kB 00:00 2024-04-02T12:17:56.232 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:17:56.235 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:17:56.235 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:17:56.266 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:17:56.266 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:17:56.361 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:17:56.487 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:17:56.898 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:17:57.157 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:17:57.157 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:57.157 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:17:57.157 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:17:57.157 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:17:57.157 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:57.336 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2 2024-04-02T12:17:57.903 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:24 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:57.998 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:17:58.052 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:58.053 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:17:58.053 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:58.123 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1 2024-04-02T12:17:58.691 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:25 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:58.790 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:17:58.843 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:58.844 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:17:58.844 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:58.918 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados 2024-04-02T12:17:59.487 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:26 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:17:59.585 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:17:59.637 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:17:59.638 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:17:59.638 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:17:59.706 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw 2024-04-02T12:18:00.273 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:27 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:00.373 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:18:00.425 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:00.426 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:18:00.426 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:18:00.497 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs 2024-04-02T12:18:01.060 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:28 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:01.159 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:18:01.210 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:01.211 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:18:01.211 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:18:01.284 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd 2024-04-02T12:18:01.851 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:28 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:01.952 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:18:02.005 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:02.006 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-04-02T12:18:02.006 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:18:02.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse 2024-04-02T12:18:02.638 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:29 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:02.790 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 90 k 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:18:02.791 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:02.792 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 90 k 2024-04-02T12:18:02.792 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 313 k 2024-04-02T12:18:02.792 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:18:03.037 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-19.0.0-2589.g38caedd8.el9.x86_64.rpm 366 kB/s | 90 kB 00:00 2024-04-02T12:18:03.038 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:18:03.038 INFO:teuthology.orchestra.run.smithi159.stdout:Total 365 kB/s | 90 kB 00:00 2024-04-02T12:18:03.038 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:18:03.046 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:18:03.046 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:18:03.103 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:18:03.103 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:18:03.247 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:18:03.365 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:03.739 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:04.000 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:04.000 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:04.000 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:18:04.000 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:18:04.000 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:04.000 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:18:04.237 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror 2024-04-02T12:18:04.826 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:31 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:04.979 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:04.979 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:04.979 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 3.4 M 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 3.4 M 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 29 M 2024-04-02T12:18:04.980 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:18:05.604 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-19.0.0-2589.g38caedd8.el9.x86_64.rpm 5.5 MB/s | 3.4 MB 00:00 2024-04-02T12:18:05.604 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:18:05.605 INFO:teuthology.orchestra.run.smithi159.stdout:Total 5.4 MB/s | 3.4 MB 00:00 2024-04-02T12:18:05.605 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:18:05.615 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:18:05.615 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:18:05.690 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:18:05.690 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:18:06.255 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:18:06.338 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:06.362 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:06.363 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:18:06.363 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T12:18:06.363 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T12:18:06.363 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T12:18:06.363 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:07.651 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:07.652 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:07.652 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:18:07.652 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:18:07.652 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:07.652 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:18:07.921 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd 2024-04-02T12:18:08.504 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:04:35 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:08.659 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 176 k 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:08.660 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-04-02T12:18:08.661 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:08.661 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 176 k 2024-04-02T12:18:08.661 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 673 k 2024-04-02T12:18:08.661 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:18:08.907 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-19.0.0-2589.g38caedd8.el9.x86_64.rpm 713 kB/s | 176 kB 00:00 2024-04-02T12:18:08.908 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:18:08.908 INFO:teuthology.orchestra.run.smithi159.stdout:Total 709 kB/s | 176 kB 00:00 2024-04-02T12:18:08.908 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:18:08.918 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:18:08.919 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:18:08.975 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:18:08.975 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:18:09.123 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:18:09.222 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:09.660 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:09.974 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:18:09.974 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:09.974 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:18:09.974 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:18:09.974 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:09.974 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:18:10.220 DEBUG:teuthology.parallel:result is None 2024-04-02T12:18:10.221 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=38caedd896c6b19ed5e43655372d95dc8e78cfda 2024-04-02T12:18:10.445 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T12:18:10.467 INFO:teuthology.orchestra.run.smithi159.stdout:19.0.0-2589.g38caedd8.el9 2024-04-02T12:18:10.468 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2589.g38caedd8.el9 2024-04-02T12:18:10.468 INFO:teuthology.task.install:The correct ceph version 19.0.0-2589.g38caedd8 is installed. 2024-04-02T12:18:10.470 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T12:18:10.470 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:10.470 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T12:18:10.534 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T12:18:10.535 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:10.535 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T12:18:10.602 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T12:18:10.669 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T12:18:10.669 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:10.669 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T12:18:10.736 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T12:18:10.803 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T12:18:10.803 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:10.803 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T12:18:10.870 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T12:18:10.937 INFO:teuthology.run_tasks:Running task ceph... 2024-04-02T12:18:11.017 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-02T12:18:11.018 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-04-02T12:18:11.044 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-02T12:18:11.045 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T12:18:11.111 INFO:tasks.ceph:Creating extra log directories... 2024-04-02T12:18:11.112 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T12:18:11.186 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-02T12:18:11.186 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-02T12:18:11.186 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514', 'branch': 'wip-matanb-crimson-testing-apr-2', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read}', 'email': None, 'first_in_suite': False, 'job_id': '7635514', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-apr-2'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '107e1624f2a9b72a56ee529969558944d24a2a75'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 223, 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '107e1624f2a9b72a56ee529969558944d24a2a75', 'targets': {'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEiGxHP0FnGWqmkNH1+sIt/K3egsTYNfXYVR2ZCqnR/vgXYWZ9TiBLbLLdTHORVMhIGS/ETnyL9HPJXo0ji0hyg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_11:03:04', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-02T12:18:11.189 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T12:18:11.233 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T12:18:11.300 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:11.300 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T12:18:11.355 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T12:18:11.355 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-04-02T12:18:11.410 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T12:18:11.410 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:18:11.410 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-02T12:18:11.410 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:18:11.411 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:18:11.411 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-02 12:18:06.643848906 +0000 2024-04-02T12:18:11.411 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-02 12:12:51.422627444 +0000 2024-04-02T12:18:11.411 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-02 12:12:51.422627444 +0000 2024-04-02T12:18:11.411 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-04-02 12:12:51.422627444 +0000 2024-04-02T12:18:11.411 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T12:18:11.475 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-02T12:18:11.475 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-02T12:18:11.475 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000184034 s, 2.8 MB/s 2024-04-02T12:18:11.476 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T12:18:11.532 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-02 12:18:06.643848906 +0000 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-02 12:12:51.779620704 +0000 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-02 12:12:51.779620704 +0000 2024-04-02T12:18:11.587 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-04-02 12:12:51.779620704 +0000 2024-04-02T12:18:11.588 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T12:18:11.653 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-02T12:18:11.653 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-02T12:18:11.653 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000194969 s, 2.6 MB/s 2024-04-02T12:18:11.654 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T12:18:11.710 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-04-02T12:18:11.764 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T12:18:11.764 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-02 12:18:06.643848906 +0000 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-02 12:12:52.127614133 +0000 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-02 12:12:52.127614133 +0000 2024-04-02T12:18:11.765 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-04-02 12:12:52.127614133 +0000 2024-04-02T12:18:11.765 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T12:18:11.831 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-02T12:18:11.831 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-02T12:18:11.831 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000245099 s, 2.1 MB/s 2024-04-02T12:18:11.832 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T12:18:11.887 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-04-02T12:18:11.942 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T12:18:11.942 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:18:11.942 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-02T12:18:11.942 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:18:11.943 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:18:11.943 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-04-02 12:18:06.644848887 +0000 2024-04-02T12:18:11.943 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-04-02 12:12:52.452607997 +0000 2024-04-02T12:18:11.943 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-04-02 12:12:52.452607997 +0000 2024-04-02T12:18:11.943 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-04-02 12:12:52.452607997 +0000 2024-04-02T12:18:11.943 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T12:18:12.007 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-04-02T12:18:12.008 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-04-02T12:18:12.008 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000185223 s, 2.8 MB/s 2024-04-02T12:18:12.009 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T12:18:12.064 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:18:12.065 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-02T12:18:12.065 INFO:tasks.ceph:Generating config... 2024-04-02T12:18:12.066 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-02T12:18:12.067 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] log to file = False 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-02T12:18:12.068 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] auth client required = none 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] auth service required = none 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] auth supported = none 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-02T12:18:12.068 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-02T12:18:12.069 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-02T12:18:12.070 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-02T12:18:12.071 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-02T12:18:12.072 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-02T12:18:12.072 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-02T12:18:12.072 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-02T12:18:12.072 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-02T12:18:12.072 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-02T12:18:12.072 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-02T12:18:12.072 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-02T12:18:12.073 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-02T12:18:12.073 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-02T12:18:12.073 INFO:tasks.ceph:Setting up mon.a... 2024-04-02T12:18:12.073 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-02T12:18:12.161 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.keyring 2024-04-02T12:18:12.164 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-02T12:18:12.216 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T12:18:12.242 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.159')] 2024-04-02T12:18:12.243 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.159', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-02T12:18:12.244 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:12.244 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T12:18:12.298 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.159 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:12.388 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:12.396+0000 7faa07230f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:12.388 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:12.396+0000 7faa07230f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:12.388 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:12.396+0000 7faa07230f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:12.388 INFO:teuthology.orchestra.run.smithi159.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:monmaptool: generated fsid c83422aa-1567-4542-acee-29d38718ee2f 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:setting min_mon_release = quincy 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:epoch 0 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:fsid c83422aa-1567-4542-acee-29d38718ee2f 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:last_changed 2024-04-02T12:18:12.397684+0000 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:created 2024-04-02T12:18:12.397684+0000 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:min_mon_release 17 (quincy) 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:election_strategy: 1 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:0: [v2:172.21.15.159:3300/0,v1:172.21.15.159:6789/0] mon.a 2024-04-02T12:18:12.389 INFO:teuthology.orchestra.run.smithi159.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-02T12:18:12.390 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T12:18:12.405 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c83422aa-1567-4542-acee-29d38718ee2f... 2024-04-02T12:18:12.406 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = none 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-04-02T12:18:12.496 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-04-02T12:18:12.497 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.498 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = 172.21.15.159 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: auth client required = none 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: auth cluster required = none 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: auth service required = none 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug asok = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug bdev = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug buffer = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug civetweb = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug client = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug compressor = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug context = 0/0 2024-04-02T12:18:12.499 INFO:teuthology.orchestra.run.smithi159.stdout: debug crush = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug crypto = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug dpdk = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug eventtrace = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug filer = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug filestore = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug finisher = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug fuse = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug heartbeatmap = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug javaclient = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug journal = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug journaler = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug kstore = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug leveldb = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug lockdep = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds balancer = 0/0 2024-04-02T12:18:12.500 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds locker = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds log = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds log expire = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug mds migrator = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug memdb = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgrc = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug objclass = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug objectcacher = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug objecter = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug optracker = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug perfcounter = 0/0 2024-04-02T12:18:12.501 INFO:teuthology.orchestra.run.smithi159.stdout: debug rados = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rbd = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rbd mirror = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rbd replay = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug refs = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rgw = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rgw lifecycle = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rgw notification = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rgw sync = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug striper = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug throttle = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug timer = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: debug tp = 0/0 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-02T12:18:12.502 INFO:teuthology.orchestra.run.smithi159.stdout: ms client mode = crc 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: ms cluster mode = crc 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: ms mon client mode = crc 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: ms mon cluster mode = crc 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: ms mon service mode = crc 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: ms service mode = crc 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = c83422aa-1567-4542-acee-29d38718ee2f 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-04-02T12:18:12.503 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-04-02T12:18:12.504 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 20 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore compression algorithm = snappy 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore compression mode = aggressive 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore rocksdb cf = False 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: crimson alien thread cpu cores = 6-7 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: crimson osd obc lru size = 10 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 20 2024-04-02T12:18:12.505 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 20 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: debug filestore = 0/0 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: debug journal = 0/0 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 0/0 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 0/0 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: err to stderr = True 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: log flush on exit = True 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: log to file = False 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: log to stderr = True 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-04-02T12:18:12.506 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: osd shutdown pgref assert = True 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.507 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 0/0 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 0/0 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 0/0 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T12:18:12.508 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default crimson = True 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-04-02T12:18:12.509 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: crimson seastar cpu cores = 0-2 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: crimson seastar cpu cores = 3-5 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: crimson seastar cpu cores = 0-2 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-04-02T12:18:12.510 INFO:teuthology.orchestra.run.smithi159.stdout: crimson seastar cpu cores = 3-5 2024-04-02T12:18:12.511 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-02T12:18:12.511 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-02T12:18:12.618 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-02T12:18:12.618 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:12.618 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-02T12:18:12.633 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:12.634 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-02T12:18:12.688 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:18:12.688 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:12.688 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T12:18:12.688 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T12:18:12.767 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:12.767 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:12.822 INFO:tasks.ceph:Setting up mon nodes... 2024-04-02T12:18:12.822 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-02T12:18:12.822 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T12:18:12.926 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T12:18:12.929 INFO:tasks.ceph:Setting up mds nodes... 2024-04-02T12:18:12.929 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-02T12:18:12.929 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T12:18:12.979 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-02T12:18:12.995 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-02T12:18:12.995 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-02T12:18:12.995 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-02T12:18:13.022 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:18:13.022 INFO:tasks.ceph:role: osd.0 2024-04-02T12:18:13.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:18:13.022 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T12:18:13.096 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T12:18:13.610 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-04-02T12:18:13.612 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-04-02T12:18:13.612 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-02T12:18:13.729 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-02T12:18:13.762 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-02T12:18:13.829 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:18:13.829 INFO:tasks.ceph:role: osd.1 2024-04-02T12:18:13.829 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:18:13.829 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T12:18:13.903 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T12:18:13.903 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T12:18:13.903 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T12:18:13.904 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T12:18:14.366 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-04-02T12:18:14.368 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-04-02T12:18:14.368 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-02T12:18:14.462 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-02T12:18:14.493 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-02T12:18:14.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:18:14.561 INFO:tasks.ceph:role: osd.2 2024-04-02T12:18:14.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:18:14.561 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T12:18:14.635 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T12:18:14.636 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T12:18:15.127 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-04-02T12:18:15.128 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-04-02T12:18:15.129 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-02T12:18:15.278 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-02T12:18:15.311 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:15.614 INFO:teuthology.orchestra.run.smithi159.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:18:15.615 INFO:teuthology.orchestra.run.smithi159.stderr:INFO 2024-04-02 12:18:15,624 seastar - Reactor backend: linux-aio 2024-04-02T12:18:15.627 INFO:teuthology.orchestra.run.smithi159.stderr:==29195==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:18:15.778 INFO:teuthology.orchestra.run.smithi159.stderr:DEBUG 2024-04-02 12:18:15,787 [shard 0:main] seastar - reactor::drain 2024-04-02T12:18:16.316 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:16.316 INFO:teuthology.orchestra.run.smithi159.stderr:Suppressions used: 2024-04-02T12:18:16.316 INFO:teuthology.orchestra.run.smithi159.stderr: count bytes template 2024-04-02T12:18:16.316 INFO:teuthology.orchestra.run.smithi159.stderr: 1 8 MallocExtension::Register 2024-04-02T12:18:16.316 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:16.316 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:16.377 INFO:teuthology.orchestra.run.smithi159.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:18:16.381 INFO:teuthology.orchestra.run.smithi159.stderr:INFO 2024-04-02 12:18:16,390 seastar - Reactor backend: linux-aio 2024-04-02T12:18:16.390 INFO:teuthology.orchestra.run.smithi159.stderr:==29193==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:18:16.488 INFO:teuthology.orchestra.run.smithi159.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9de8c9 0xb8b1390 0xb8b290c 0xb8b2b22 0xb8b2c78 0xb8b3141 0x54daf 0x105b10 0x2a517 0xe0e5c 0x34cf0c2 0x34cf4be 0x34cf5d7 0x5da9d34 0x5bee4a7 0x5c0956b 0x6ee1bf4 0x6ef33c7 0x6ef4864 0x6ef48e9 0x6ef49eb 0x6ef4ae3 0x6ef4cf0 0x6ef4ed6 0x6ef5225 0x6ef55c9 0x6ef57f9 0x6ef5a63 0x6ef5d30 0x6ef5f4e 0x6ef6006 0x6ef60cc 0x6ef61ad 0x5a54aa8 0x5a54bb2 0x5a54c6e 0x5a54d2a 0x5a54de6 0x5a54f33 0x5a550be 0x5a5547f 0x5a71efc 0xb8c4015 0xb8de2af 0xb97f2ad 0xb9808fb 0xb65d063 0xb65d9df 0x368b58a 0x3feaf 0x3ff5f 0x3477444 2024-04-02T12:18:16.488 INFO:teuthology.orchestra.run.smithi159.stderr:kernel callstack: 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stderr:Suppressions used: 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stderr: count bytes template 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stderr: 1 8 MallocExtension::Register 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:21.042 INFO:teuthology.orchestra.run.smithi159.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid c83422aa-1567-4542-acee-29d38718ee2f 2024-04-02T12:18:21.065 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T12:18:21.092 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:21.380 INFO:teuthology.orchestra.run.smithi159.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:18:21.381 INFO:teuthology.orchestra.run.smithi159.stderr:INFO 2024-04-02 12:18:21,391 seastar - Reactor backend: linux-aio 2024-04-02T12:18:21.386 INFO:teuthology.orchestra.run.smithi159.stderr:==29377==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:18:21.518 INFO:teuthology.orchestra.run.smithi159.stderr:DEBUG 2024-04-02 12:18:21,527 [shard 0:main] seastar - reactor::drain 2024-04-02T12:18:22.045 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:22.045 INFO:teuthology.orchestra.run.smithi159.stderr:Suppressions used: 2024-04-02T12:18:22.045 INFO:teuthology.orchestra.run.smithi159.stderr: count bytes template 2024-04-02T12:18:22.045 INFO:teuthology.orchestra.run.smithi159.stderr: 1 8 MallocExtension::Register 2024-04-02T12:18:22.045 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:22.045 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:22.106 INFO:teuthology.orchestra.run.smithi159.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:18:22.109 INFO:teuthology.orchestra.run.smithi159.stderr:INFO 2024-04-02 12:18:22,119 seastar - Reactor backend: linux-aio 2024-04-02T12:18:22.124 INFO:teuthology.orchestra.run.smithi159.stderr:==29375==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stderr:Suppressions used: 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stderr: count bytes template 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stderr: 1 8 MallocExtension::Register 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:26.757 INFO:teuthology.orchestra.run.smithi159.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid c83422aa-1567-4542-acee-29d38718ee2f 2024-04-02T12:18:26.778 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T12:18:26.805 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:27.093 INFO:teuthology.orchestra.run.smithi159.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:18:27.094 INFO:teuthology.orchestra.run.smithi159.stderr:INFO 2024-04-02 12:18:27,104 seastar - Reactor backend: linux-aio 2024-04-02T12:18:27.099 INFO:teuthology.orchestra.run.smithi159.stderr:==29559==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:18:27.206 INFO:teuthology.orchestra.run.smithi159.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9de8c9 0xb8b1390 0xb8b290c 0xb8b2b22 0xb8b2c78 0xb8b3141 0x54daf 0x3491afc 0x349c85a 0x5dcdc2f 0x5dce034 0x5dce416 0x5e2a3a4 0x5da8f20 0x5da90fa 0x5da9686 0x5bee4a7 0x5c0956b 0x6ee1bf4 0x6ef33c7 0x6ef4864 0x6ef48e9 0x6ef49eb 0x6ef4ae3 0x6ef4cf0 0x6ef4ed6 0x6ef5225 0x6ef55c9 0x6ef57f9 0x6ef5a63 0x6ef5d30 0x6ef5f4e 0x6ef6006 0x6ef60cc 0x6ef61ad 0x5a54aa8 0x5a54bb2 0x5a54c6e 0x5a54d2a 0x5a54de6 0x5a54f33 0x5a550be 0x5a5547f 0x5a71efc 0xb8c4015 0xb8de2af 0xb97f2ad 0xb9808fb 0xb65d063 0xb65d9df 0x37ea1ce 0x37ead27 0x368ad7a 0x3feaf 0x3ff5f 0x3477444 2024-04-02T12:18:27.206 INFO:teuthology.orchestra.run.smithi159.stderr:kernel callstack: 2024-04-02T12:18:27.235 INFO:teuthology.orchestra.run.smithi159.stderr:DEBUG 2024-04-02 12:18:27,245 [shard 0:main] seastar - reactor::drain 2024-04-02T12:18:27.764 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:27.765 INFO:teuthology.orchestra.run.smithi159.stderr:Suppressions used: 2024-04-02T12:18:27.765 INFO:teuthology.orchestra.run.smithi159.stderr: count bytes template 2024-04-02T12:18:27.765 INFO:teuthology.orchestra.run.smithi159.stderr: 1 8 MallocExtension::Register 2024-04-02T12:18:27.765 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:27.765 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:27.819 INFO:teuthology.orchestra.run.smithi159.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:18:27.822 INFO:teuthology.orchestra.run.smithi159.stderr:INFO 2024-04-02 12:18:27,832 seastar - Reactor backend: linux-aio 2024-04-02T12:18:27.836 INFO:teuthology.orchestra.run.smithi159.stderr:==29557==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:18:32.472 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:32.472 INFO:teuthology.orchestra.run.smithi159.stderr:Suppressions used: 2024-04-02T12:18:32.472 INFO:teuthology.orchestra.run.smithi159.stderr: count bytes template 2024-04-02T12:18:32.472 INFO:teuthology.orchestra.run.smithi159.stderr: 1 8 MallocExtension::Register 2024-04-02T12:18:32.472 INFO:teuthology.orchestra.run.smithi159.stderr:----------------------------------------------------- 2024-04-02T12:18:32.472 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:32.473 INFO:teuthology.orchestra.run.smithi159.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid c83422aa-1567-4542-acee-29d38718ee2f 2024-04-02T12:18:32.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T12:18:32.521 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-02T12:18:32.522 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:32.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-02T12:18:32.589 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:32.589 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-02T12:18:32.656 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:32.656 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-02T12:18:32.722 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:32.723 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-02T12:18:32.789 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:32.790 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-02T12:18:32.845 INFO:tasks.ceph:Adding keys to all mons... 2024-04-02T12:18:32.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T12:18:32.910 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-04-02T12:18:32.910 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCE9wtm5W/CNxAAh0pgjdL1mOxFgpZwnSFXeg== 2024-04-02T12:18:32.910 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-04-02T12:18:32.910 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCI9wtmpahbIRAAPFAmtCIXE8QVu/bdEc7Ajg== 2024-04-02T12:18:32.910 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-04-02T12:18:32.911 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCO9wtm67rIEBAAIFoLVtFNfCHiORjiuXoPxA== 2024-04-02T12:18:32.911 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-04-02T12:18:32.911 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCT9wtmXc87OxAAiHFo/djCgVvv/X6+hAvvXw== 2024-04-02T12:18:32.911 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-04-02T12:18:32.911 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQCE9wtmfOPwOhAAD+HT3i1iar9uowRQCT/Tfw== 2024-04-02T12:18:32.912 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T12:18:33.015 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T12:18:33.077 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T12:18:33.140 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T12:18:33.202 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T12:18:33.267 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-02T12:18:33.268 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-02T12:18:33.297 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T12:18:33.407 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:33.416+0000 7f8a4d82aa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.408 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:33.416+0000 7f8a4d82aa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-02T12:18:33.447 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:18:33.502 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-02T12:18:33.502 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-02T12:18:33.502 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-02T12:18:33.545 INFO:tasks.ceph.mon.a:Started 2024-04-02T12:18:33.545 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-02T12:18:33.545 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-02T12:18:33.545 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-02T12:18:33.547 INFO:tasks.ceph.mgr.x:Started 2024-04-02T12:18:33.547 INFO:tasks.ceph:Setting crush tunables to default 2024-04-02T12:18:33.547 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd crush tunables default 2024-04-02T12:18:33.630 INFO:tasks.ceph.mon.a.smithi159.stderr:2024-04-02T12:18:33.638+0000 7f3c13abaa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.630 INFO:tasks.ceph.mon.a.smithi159.stderr:2024-04-02T12:18:33.638+0000 7f3c13abaa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.630 INFO:tasks.ceph.mon.a.smithi159.stderr:2024-04-02T12:18:33.638+0000 7f3c13abaa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.630 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:33.639+0000 7f0fab971000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.661 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:33.665+0000 7f9736f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.663 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:33.672+0000 7f9736f1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.732 INFO:teuthology.orchestra.run.smithi159.stderr:adjusted tunables profile to default 2024-04-02T12:18:33.743 INFO:tasks.ceph:check_enable_crimson: True 2024-04-02T12:18:33.744 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-02T12:18:33.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-02T12:18:33.833 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:33.841+0000 7f0fab971000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.833 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:33.842+0000 7f0fab971000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.851 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:33.855+0000 7f4667954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.853 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:33.862+0000 7f4667954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:33.932 INFO:tasks.ceph.mon.a.smithi159.stderr:2024-04-02T12:18:33.941+0000 7f3c0a7f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-02T12:18:33.932 INFO:tasks.ceph.mon.a.smithi159.stderr:Please be aware that this feature is experimental, untested, 2024-04-02T12:18:33.932 INFO:tasks.ceph.mon.a.smithi159.stderr:unsupported, and may result in data corruption, data loss, 2024-04-02T12:18:33.932 INFO:tasks.ceph.mon.a.smithi159.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-02T12:18:33.933 INFO:tasks.ceph.mon.a.smithi159.stderr:feature with important data. 2024-04-02T12:18:33.933 INFO:tasks.ceph.mon.a.smithi159.stderr: 2024-04-02T12:18:34.294 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:34.303+0000 7f0fab971000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T12:18:34.346 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:34.355+0000 7f0fab971000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T12:18:34.745 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-02T12:18:34.745 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:34.745 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-02T12:18:34.772 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:34.772 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-02T12:18:34.839 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:34.839 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-02T12:18:34.878 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:34.887+0000 7f0fab971000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T12:18:34.904 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 5ae8342e-b75f-420a-adc2-2d4a5d3a03f0 0 2024-04-02T12:18:34.934 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:34.943+0000 7f0fab971000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T12:18:34.981 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:34.989+0000 7f0fab971000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T12:18:35.037 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.046+0000 7f0fab971000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T12:18:35.044 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.048+0000 7f93b1b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.046 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.055+0000 7f93b1b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.081 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.090+0000 7f0fab971000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T12:18:35.125 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-04-02T12:18:35.136 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new a044d59e-52bd-4f13-bc59-8b8413d69da6 1 2024-04-02T12:18:35.203 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.212+0000 7f0fab971000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T12:18:35.247 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.251+0000 7f0741b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.249 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.258+0000 7f0741b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.277 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.286+0000 7f0fab971000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T12:18:35.323 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.332+0000 7f0fab971000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T12:18:35.328 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-04-02T12:18:35.339 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 3bc80b03-bec4-4913-8efa-10cb769945fa 2 2024-04-02T12:18:35.379 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.388+0000 7f0fab971000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T12:18:35.429 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.438+0000 7f0fab971000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T12:18:35.450 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.453+0000 7fb41cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.452 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.461+0000 7fb41cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.532 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-04-02T12:18:35.543 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-02T12:18:35.615 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-02T12:18:35.615 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-02T12:18:35.657 INFO:tasks.ceph.osd.0:Started 2024-04-02T12:18:35.658 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-02T12:18:35.689 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-02T12:18:35.690 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-02T12:18:35.731 INFO:tasks.ceph.osd.1:Started 2024-04-02T12:18:35.732 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-02T12:18:35.764 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-02T12:18:35.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-02T12:18:35.806 INFO:tasks.ceph.osd.2:Started 2024-04-02T12:18:35.807 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T12:18:35.842 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:35.851+0000 7f0fab971000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T12:18:35.975 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.979+0000 7f0d6086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:35.977 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:35.986+0000 7f0d6086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:36.027 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.036+0000 7f0fab971000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T12:18:36.049 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:36.049 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":6,"fsid":"c83422aa-1567-4542-acee-29d38718ee2f","created":"2024-04-02T12:18:33.656844+0000","modified":"2024-04-02T12:18:35.541181+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T12:18:35.541181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5ae8342e-b75f-420a-adc2-2d4a5d3a03f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a044d59e-52bd-4f13-bc59-8b8413d69da6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3bc80b03-bec4-4913-8efa-10cb769945fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:18:36.065 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-02T12:18:36.065 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-02T12:18:36.147 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.155+0000 7f0fab971000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T12:18:36.194 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.203+0000 7f0fab971000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T12:18:36.313 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.321+0000 7f0fab971000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T12:18:36.369 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T12:18:36.415 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.424+0000 7f0fab971000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T12:18:36.474 INFO:teuthology.misc.health.smithi159.stderr:2024-04-02T12:18:36.479+0000 7f1e46954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:36.476 INFO:teuthology.misc.health.smithi159.stderr:2024-04-02T12:18:36.485+0000 7f1e46954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:36.544 INFO:teuthology.misc.health.smithi159.stdout: 2024-04-02T12:18:36.544 INFO:teuthology.misc.health.smithi159.stdout:{"epoch":6,"fsid":"c83422aa-1567-4542-acee-29d38718ee2f","created":"2024-04-02T12:18:33.656844+0000","modified":"2024-04-02T12:18:35.541181+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T12:18:35.541181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5ae8342e-b75f-420a-adc2-2d4a5d3a03f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a044d59e-52bd-4f13-bc59-8b8413d69da6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3bc80b03-bec4-4913-8efa-10cb769945fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:18:36.555 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-02T12:18:36.587 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.596+0000 7f0fab971000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T12:18:36.634 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.642+0000 7f0fab971000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T12:18:36.803 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:36.812+0000 7f0fab971000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T12:18:37.830 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:37.839+0000 7f0fab971000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T12:18:38.286 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:38.295+0000 7f0fab971000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T12:18:38.369 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:38.377+0000 7f0fab971000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T12:18:38.513 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:38.522+0000 7f0fab971000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T12:18:39.522 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-04-02T12:18:39.531+0000 7f0f4ab22640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-02T12:18:42.859 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T12:18:42.960 INFO:teuthology.misc.health.smithi159.stderr:2024-04-02T12:18:42.964+0000 7fac356ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:42.961 INFO:teuthology.misc.health.smithi159.stderr:2024-04-02T12:18:42.970+0000 7fac356ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:43.146 INFO:teuthology.misc.health.smithi159.stdout: 2024-04-02T12:18:43.146 INFO:teuthology.misc.health.smithi159.stdout:{"epoch":9,"fsid":"c83422aa-1567-4542-acee-29d38718ee2f","created":"2024-04-02T12:18:33.656844+0000","modified":"2024-04-02T12:18:42.537529+0000","last_up_change":"2024-04-02T12:18:42.537529+0000","last_in_change":"2024-04-02T12:18:35.541181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5ae8342e-b75f-420a-adc2-2d4a5d3a03f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2744620640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":2744620640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2744620640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":2744620640}]},"public_addr":"172.21.15.159:6806/2744620640","cluster_addr":"172.21.15.159:6805/2744620640","heartbeat_back_addr":"172.21.15.159:6802/2744620640","heartbeat_front_addr":"172.21.15.159:6807/2744620640","state":["exists","up"]},{"osd":1,"uuid":"a044d59e-52bd-4f13-bc59-8b8413d69da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":812570545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":812570545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":812570545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":812570545}]},"public_addr":"172.21.15.159:6803/812570545","cluster_addr":"172.21.15.159:6804/812570545","heartbeat_back_addr":"172.21.15.159:6812/812570545","heartbeat_front_addr":"172.21.15.159:6811/812570545","state":["exists","up"]},{"osd":2,"uuid":"3bc80b03-bec4-4913-8efa-10cb769945fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":3162026306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":3162026306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":3162026306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3162026306}]},"public_addr":"172.21.15.159:6809/3162026306","cluster_addr":"172.21.15.159:6808/3162026306","heartbeat_back_addr":"172.21.15.159:6813/3162026306","heartbeat_front_addr":"172.21.15.159:6814/3162026306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:18:43.158 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-02T12:18:43.158 INFO:tasks.ceph:Creating RBD pool 2024-04-02T12:18:43.158 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-02T12:18:43.269 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:43.271+0000 7f24b0eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:43.271 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:43.279+0000 7f24b0eed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:43.540 INFO:teuthology.orchestra.run.smithi159.stderr:pool 'rbd' created 2024-04-02T12:18:43.559 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-02T12:18:43.681 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:43.681+0000 7f31f422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:43.683 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:43.692+0000 7f31f422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:44.541 INFO:teuthology.orchestra.run.smithi159.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-02T12:18:44.563 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-02T12:18:44.563 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T12:18:44.564 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-02T12:18:44.564 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-02T12:18:44.564 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-02T12:18:44.711 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:44.716+0000 7f07dd95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:44.713 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:44.721+0000 7f07dd95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:44.949 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:44.960 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6800","nonce":4257305068},{"type":"v1","addr":"172.21.15.159:6801","nonce":4257305068}]},"active_addr":"172.21.15.159:6801/4257305068","active_change":"2024-04-02T12:18:38.524224+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1842168054}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":4076924830}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.159:0","nonce":1297390419}]}]} 2024-04-02T12:18:44.964 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-02T12:18:44.964 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-02T12:18:44.964 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T12:18:45.075 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.080+0000 7f984e75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.077 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.086+0000 7f984e75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.263 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:45.263 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":11,"fsid":"c83422aa-1567-4542-acee-29d38718ee2f","created":"2024-04-02T12:18:33.656844+0000","modified":"2024-04-02T12:18:44.550156+0000","last_up_change":"2024-04-02T12:18:42.537529+0000","last_in_change":"2024-04-02T12:18:35.541181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-02T12:18:43.468430+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-02T12:18:44.544811+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ae8342e-b75f-420a-adc2-2d4a5d3a03f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2744620640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":2744620640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2744620640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":2744620640}]},"public_addr":"172.21.15.159:6806/2744620640","cluster_addr":"172.21.15.159:6805/2744620640","heartbeat_back_addr":"172.21.15.159:6802/2744620640","heartbeat_front_addr":"172.21.15.159:6807/2744620640","state":["exists","up"]},{"osd":1,"uuid":"a044d59e-52bd-4f13-bc59-8b8413d69da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":812570545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":812570545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":812570545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":812570545}]},"public_addr":"172.21.15.159:6803/812570545","cluster_addr":"172.21.15.159:6804/812570545","heartbeat_back_addr":"172.21.15.159:6812/812570545","heartbeat_front_addr":"172.21.15.159:6811/812570545","state":["exists","up"]},{"osd":2,"uuid":"3bc80b03-bec4-4913-8efa-10cb769945fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":3162026306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":3162026306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":3162026306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3162026306}]},"public_addr":"172.21.15.159:6809/3162026306","cluster_addr":"172.21.15.159:6808/3162026306","heartbeat_back_addr":"172.21.15.159:6813/3162026306","heartbeat_front_addr":"172.21.15.159:6814/3162026306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:18:45.276 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-02T12:18:45.276 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T12:18:45.388 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.393+0000 7f6e81664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.390 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.399+0000 7f6e81664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.595 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:45.596 INFO:teuthology.orchestra.run.smithi159.stdout:{"epoch":12,"fsid":"c83422aa-1567-4542-acee-29d38718ee2f","created":"2024-04-02T12:18:33.656844+0000","modified":"2024-04-02T12:18:45.554553+0000","last_up_change":"2024-04-02T12:18:42.537529+0000","last_in_change":"2024-04-02T12:18:35.541181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-02T12:18:43.468430+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-02T12:18:44.544811+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ae8342e-b75f-420a-adc2-2d4a5d3a03f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6806","nonce":2744620640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6805","nonce":2744620640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6802","nonce":2744620640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6807","nonce":2744620640}]},"public_addr":"172.21.15.159:6806/2744620640","cluster_addr":"172.21.15.159:6805/2744620640","heartbeat_back_addr":"172.21.15.159:6802/2744620640","heartbeat_front_addr":"172.21.15.159:6807/2744620640","state":["exists","up"]},{"osd":1,"uuid":"a044d59e-52bd-4f13-bc59-8b8413d69da6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6803","nonce":812570545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6804","nonce":812570545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6812","nonce":812570545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6811","nonce":812570545}]},"public_addr":"172.21.15.159:6803/812570545","cluster_addr":"172.21.15.159:6804/812570545","heartbeat_back_addr":"172.21.15.159:6812/812570545","heartbeat_front_addr":"172.21.15.159:6811/812570545","state":["exists","up"]},{"osd":2,"uuid":"3bc80b03-bec4-4913-8efa-10cb769945fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6809","nonce":3162026306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6808","nonce":3162026306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6813","nonce":3162026306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.159:6814","nonce":3162026306}]},"public_addr":"172.21.15.159:6809/3162026306","cluster_addr":"172.21.15.159:6808/3162026306","heartbeat_back_addr":"172.21.15.159:6813/3162026306","heartbeat_front_addr":"172.21.15.159:6814/3162026306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:18:45.614 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-02T12:18:45.615 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-02T12:18:45.615 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-02T12:18:45.748 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.749+0000 7fb7116ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.750 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.759+0000 7fb7116ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.761 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.763+0000 7f1000be2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.763 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.772+0000 7f1000be2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.771 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.772+0000 7fd40871d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.783+0000 7fd40871d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.785 INFO:teuthology.orchestra.run.smithi159.stdout:38654705667 2024-04-02T12:18:45.786 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:18:45.797 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2024-04-02T12:18:45.798 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:18:45.815 INFO:teuthology.orchestra.run.smithi159.stdout:38654705667 2024-04-02T12:18:45.816 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T12:18:45.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.942+0000 7f2e1ccad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.950+0000 7f2e1ccad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.943+0000 7f39e3464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:45.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:45.953+0000 7f39e3464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:46.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:46.004+0000 7f36dbf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:46.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:46.011+0000 7f36dbf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:46.203 INFO:teuthology.orchestra.run.smithi159.stdout:38654705667 2024-04-02T12:18:46.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2024-04-02T12:18:46.220 DEBUG:teuthology.parallel:result is None 2024-04-02T12:18:46.237 INFO:teuthology.orchestra.run.smithi159.stdout:38654705667 2024-04-02T12:18:46.242 INFO:teuthology.orchestra.run.smithi159.stdout:34359738372 2024-04-02T12:18:46.253 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2024-04-02T12:18:46.253 DEBUG:teuthology.parallel:result is None 2024-04-02T12:18:46.259 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2024-04-02T12:18:46.260 DEBUG:teuthology.parallel:result is None 2024-04-02T12:18:46.260 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T12:18:46.260 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T12:18:46.444 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:46.449+0000 7f3b60b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:46.447 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:46.455+0000 7f3b60b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:46.682 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:46.683 INFO:teuthology.orchestra.run.smithi159.stderr:dumped all 2024-04-02T12:18:46.699 INFO:teuthology.orchestra.run.smithi159.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-02T12:18:46.563439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131584,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":32,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80756,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034764,"statfs":{"total":289910292480,"available":289827598336,"internally_reserved":0,"allocated":290816,"data_stored":63625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.750883"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.584212+0000","last_change":"2024-04-02T12:18:44.645480+0000","last_active":"2024-04-02T12:18:45.584212+0000","last_peered":"2024-04-02T12:18:45.584212+0000","last_clean":"2024-04-02T12:18:45.584212+0000","last_became_active":"2024-04-02T12:18:44.643066+0000","last_became_peered":"2024-04-02T12:18:44.643066+0000","last_unstale":"2024-04-02T12:18:45.584212+0000","last_undegraded":"2024-04-02T12:18:45.584212+0000","last_fullsized":"2024-04-02T12:18:45.584212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.581556+0000","last_change":"2024-04-02T12:18:44.667415+0000","last_active":"2024-04-02T12:18:45.581556+0000","last_peered":"2024-04-02T12:18:45.581556+0000","last_clean":"2024-04-02T12:18:45.581556+0000","last_became_active":"2024-04-02T12:18:44.662793+0000","last_became_peered":"2024-04-02T12:18:44.662793+0000","last_unstale":"2024-04-02T12:18:45.581556+0000","last_undegraded":"2024-04-02T12:18:45.581556+0000","last_fullsized":"2024-04-02T12:18:45.581556+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.567818+0000","last_change":"2024-04-02T12:18:44.660381+0000","last_active":"2024-04-02T12:18:45.567818+0000","last_peered":"2024-04-02T12:18:45.567818+0000","last_clean":"2024-04-02T12:18:45.567818+0000","last_became_active":"2024-04-02T12:18:44.658175+0000","last_became_peered":"2024-04-02T12:18:44.658175+0000","last_unstale":"2024-04-02T12:18:45.567818+0000","last_undegraded":"2024-04-02T12:18:45.567818+0000","last_fullsized":"2024-04-02T12:18:45.567818+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.584002+0000","last_change":"2024-04-02T12:18:44.698413+0000","last_active":"2024-04-02T12:18:45.584002+0000","last_peered":"2024-04-02T12:18:45.584002+0000","last_clean":"2024-04-02T12:18:45.584002+0000","last_became_active":"2024-04-02T12:18:44.696342+0000","last_became_peered":"2024-04-02T12:18:44.696342+0000","last_unstale":"2024-04-02T12:18:45.584002+0000","last_undegraded":"2024-04-02T12:18:45.584002+0000","last_fullsized":"2024-04-02T12:18:45.584002+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'12","reported_seq":27,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.781420+0000","last_change":"2024-04-02T12:18:45.604339+0000","last_active":"2024-04-02T12:18:45.781420+0000","last_peered":"2024-04-02T12:18:45.781420+0000","last_clean":"2024-04-02T12:18:45.781420+0000","last_became_active":"2024-04-02T12:18:45.600979+0000","last_became_peered":"2024-04-02T12:18:45.600979+0000","last_unstale":"2024-04-02T12:18:45.781420+0000","last_undegraded":"2024-04-02T12:18:45.781420+0000","last_fullsized":"2024-04-02T12:18:45.781420+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:44.550156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:44.550156+0000","last_clean_scrub_stamp":"2024-04-02T12:18:44.550156+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131584,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":32,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.584548+0000","last_change":"2024-04-02T12:18:44.656484+0000","last_active":"2024-04-02T12:18:45.584548+0000","last_peered":"2024-04-02T12:18:45.584548+0000","last_clean":"2024-04-02T12:18:45.584548+0000","last_became_active":"2024-04-02T12:18:44.654230+0000","last_became_peered":"2024-04-02T12:18:44.654230+0000","last_unstale":"2024-04-02T12:18:45.584548+0000","last_undegraded":"2024-04-02T12:18:45.584548+0000","last_fullsized":"2024-04-02T12:18:45.584548+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-02T12:18:44.670429+0000","last_change":"2024-04-02T12:18:44.674966+0000","last_active":"2024-04-02T12:18:44.670429+0000","last_peered":"2024-04-02T12:18:44.670429+0000","last_clean":"2024-04-02T12:18:43.547103+0000","last_became_active":"2024-04-02T12:18:44.664716+0000","last_became_peered":"2024-04-02T12:18:44.664716+0000","last_unstale":"2024-04-02T12:18:44.670429+0000","last_undegraded":"2024-04-02T12:18:44.670429+0000","last_fullsized":"2024-04-02T12:18:44.670429+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.565629+0000","last_change":"2024-04-02T12:18:44.637012+0000","last_active":"2024-04-02T12:18:45.565629+0000","last_peered":"2024-04-02T12:18:45.565629+0000","last_clean":"2024-04-02T12:18:45.565629+0000","last_became_active":"2024-04-02T12:18:44.634348+0000","last_became_peered":"2024-04-02T12:18:44.634348+0000","last_unstale":"2024-04-02T12:18:45.565629+0000","last_undegraded":"2024-04-02T12:18:45.565629+0000","last_fullsized":"2024-04-02T12:18:45.565629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.582154+0000","last_change":"2024-04-02T12:18:44.686885+0000","last_active":"2024-04-02T12:18:45.582154+0000","last_peered":"2024-04-02T12:18:45.582154+0000","last_clean":"2024-04-02T12:18:45.582154+0000","last_became_active":"2024-04-02T12:18:44.681211+0000","last_became_peered":"2024-04-02T12:18:44.681211+0000","last_unstale":"2024-04-02T12:18:45.582154+0000","last_undegraded":"2024-04-02T12:18:45.582154+0000","last_fullsized":"2024-04-02T12:18:45.582154+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":131584,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":32,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":131584,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":23389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":20118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":20118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":131584,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T12:18:46.700 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T12:18:46.847 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:46.851+0000 7fb1c3264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:46.849 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:46.858+0000 7fb1c3264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:47.037 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:47.037 INFO:teuthology.orchestra.run.smithi159.stderr:dumped all 2024-04-02T12:18:47.050 INFO:teuthology.orchestra.run.smithi159.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-02T12:18:46.563439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131584,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":32,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80756,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034764,"statfs":{"total":289910292480,"available":289827598336,"internally_reserved":0,"allocated":290816,"data_stored":63625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.750883"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.584212+0000","last_change":"2024-04-02T12:18:44.645480+0000","last_active":"2024-04-02T12:18:45.584212+0000","last_peered":"2024-04-02T12:18:45.584212+0000","last_clean":"2024-04-02T12:18:45.584212+0000","last_became_active":"2024-04-02T12:18:44.643066+0000","last_became_peered":"2024-04-02T12:18:44.643066+0000","last_unstale":"2024-04-02T12:18:45.584212+0000","last_undegraded":"2024-04-02T12:18:45.584212+0000","last_fullsized":"2024-04-02T12:18:45.584212+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.581556+0000","last_change":"2024-04-02T12:18:44.667415+0000","last_active":"2024-04-02T12:18:45.581556+0000","last_peered":"2024-04-02T12:18:45.581556+0000","last_clean":"2024-04-02T12:18:45.581556+0000","last_became_active":"2024-04-02T12:18:44.662793+0000","last_became_peered":"2024-04-02T12:18:44.662793+0000","last_unstale":"2024-04-02T12:18:45.581556+0000","last_undegraded":"2024-04-02T12:18:45.581556+0000","last_fullsized":"2024-04-02T12:18:45.581556+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.567818+0000","last_change":"2024-04-02T12:18:44.660381+0000","last_active":"2024-04-02T12:18:45.567818+0000","last_peered":"2024-04-02T12:18:45.567818+0000","last_clean":"2024-04-02T12:18:45.567818+0000","last_became_active":"2024-04-02T12:18:44.658175+0000","last_became_peered":"2024-04-02T12:18:44.658175+0000","last_unstale":"2024-04-02T12:18:45.567818+0000","last_undegraded":"2024-04-02T12:18:45.567818+0000","last_fullsized":"2024-04-02T12:18:45.567818+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.584002+0000","last_change":"2024-04-02T12:18:44.698413+0000","last_active":"2024-04-02T12:18:45.584002+0000","last_peered":"2024-04-02T12:18:45.584002+0000","last_clean":"2024-04-02T12:18:45.584002+0000","last_became_active":"2024-04-02T12:18:44.696342+0000","last_became_peered":"2024-04-02T12:18:44.696342+0000","last_unstale":"2024-04-02T12:18:45.584002+0000","last_undegraded":"2024-04-02T12:18:45.584002+0000","last_fullsized":"2024-04-02T12:18:45.584002+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'12","reported_seq":27,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.781420+0000","last_change":"2024-04-02T12:18:45.604339+0000","last_active":"2024-04-02T12:18:45.781420+0000","last_peered":"2024-04-02T12:18:45.781420+0000","last_clean":"2024-04-02T12:18:45.781420+0000","last_became_active":"2024-04-02T12:18:45.600979+0000","last_became_peered":"2024-04-02T12:18:45.600979+0000","last_unstale":"2024-04-02T12:18:45.781420+0000","last_undegraded":"2024-04-02T12:18:45.781420+0000","last_fullsized":"2024-04-02T12:18:45.781420+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:44.550156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:44.550156+0000","last_clean_scrub_stamp":"2024-04-02T12:18:44.550156+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131584,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":32,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.584548+0000","last_change":"2024-04-02T12:18:44.656484+0000","last_active":"2024-04-02T12:18:45.584548+0000","last_peered":"2024-04-02T12:18:45.584548+0000","last_clean":"2024-04-02T12:18:45.584548+0000","last_became_active":"2024-04-02T12:18:44.654230+0000","last_became_peered":"2024-04-02T12:18:44.654230+0000","last_unstale":"2024-04-02T12:18:45.584548+0000","last_undegraded":"2024-04-02T12:18:45.584548+0000","last_fullsized":"2024-04-02T12:18:45.584548+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-02T12:18:44.670429+0000","last_change":"2024-04-02T12:18:44.674966+0000","last_active":"2024-04-02T12:18:44.670429+0000","last_peered":"2024-04-02T12:18:44.670429+0000","last_clean":"2024-04-02T12:18:43.547103+0000","last_became_active":"2024-04-02T12:18:44.664716+0000","last_became_peered":"2024-04-02T12:18:44.664716+0000","last_unstale":"2024-04-02T12:18:44.670429+0000","last_undegraded":"2024-04-02T12:18:44.670429+0000","last_fullsized":"2024-04-02T12:18:44.670429+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.565629+0000","last_change":"2024-04-02T12:18:44.637012+0000","last_active":"2024-04-02T12:18:45.565629+0000","last_peered":"2024-04-02T12:18:45.565629+0000","last_clean":"2024-04-02T12:18:45.565629+0000","last_became_active":"2024-04-02T12:18:44.634348+0000","last_became_peered":"2024-04-02T12:18:44.634348+0000","last_unstale":"2024-04-02T12:18:45.565629+0000","last_undegraded":"2024-04-02T12:18:45.565629+0000","last_fullsized":"2024-04-02T12:18:45.565629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T12:18:45.582154+0000","last_change":"2024-04-02T12:18:44.686885+0000","last_active":"2024-04-02T12:18:45.582154+0000","last_peered":"2024-04-02T12:18:45.582154+0000","last_clean":"2024-04-02T12:18:45.582154+0000","last_became_active":"2024-04-02T12:18:44.681211+0000","last_became_peered":"2024-04-02T12:18:44.681211+0000","last_unstale":"2024-04-02T12:18:45.582154+0000","last_undegraded":"2024-04-02T12:18:45.582154+0000","last_fullsized":"2024-04-02T12:18:45.582154+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:18:43.547103+0000","last_clean_scrub_stamp":"2024-04-02T12:18:43.547103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":131584,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":32,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":131584,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":23389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":20118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":20118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":131584,"data_compressed":6340,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T12:18:47.050 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T12:18:47.051 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T12:18:47.051 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-02T12:18:47.051 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T12:18:47.161 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:47.167+0000 7fbe17f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:47.163 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:18:47.172+0000 7fbe17f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:18:47.386 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:47.386 INFO:teuthology.orchestra.run.smithi159.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T12:18:47.399 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-02T12:18:47.399 INFO:teuthology.run_tasks:Running task cbt... 2024-04-02T12:18:47.413 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi159.front.sepia.ceph.com', 'osds': ['ubuntu@smithi159.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi159.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi159.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi159.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-02T12:18:47.413 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-02T12:18:47.430 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:18:47.431 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-02T12:18:47.484 DEBUG:teuthology.orchestra.run.smithi159:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-02T12:18:47.539 INFO:teuthology.orchestra.run.smithi159.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-02T12:18:47.916 INFO:tasks.cbt:Installing collectl 2024-04-02T12:18:47.916 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-02T12:18:48.019 INFO:teuthology.orchestra.run.smithi159.stderr:--2024-04-02 12:18:48-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-02T12:18:48.177 INFO:teuthology.orchestra.run.smithi159.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2024-04-02T12:18:48.186 INFO:teuthology.orchestra.run.smithi159.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-04-02T12:18:48.410 INFO:teuthology.orchestra.run.smithi159.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-02T12:18:48.411 INFO:teuthology.orchestra.run.smithi159.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmC_eoCUeHwbM7rrJA83Fqfp9JtBRkCGozKwfeX8IkiTBupRFfEwMo3MIrN1W_Pg8gdbsySLpju32c3jWf-vII4jhjeg%3D%3D&use_mirror=gigenet&r= [following] 2024-04-02T12:18:48.411 INFO:teuthology.orchestra.run.smithi159.stderr:--2024-04-02 12:18:48-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmC_eoCUeHwbM7rrJA83Fqfp9JtBRkCGozKwfeX8IkiTBupRFfEwMo3MIrN1W_Pg8gdbsySLpju32c3jWf-vII4jhjeg%3D%3D&use_mirror=gigenet&r= 2024-04-02T12:18:48.421 INFO:teuthology.orchestra.run.smithi159.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-02T12:18:48.489 INFO:teuthology.orchestra.run.smithi159.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-02T12:18:48.754 INFO:teuthology.orchestra.run.smithi159.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-02T12:18:48.754 INFO:teuthology.orchestra.run.smithi159.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-02T12:18:48.754 INFO:teuthology.orchestra.run.smithi159.stderr:--2024-04-02 12:18:48-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-02T12:18:48.767 INFO:teuthology.orchestra.run.smithi159.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2024-04-02T12:18:48.801 INFO:teuthology.orchestra.run.smithi159.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2024-04-02T12:18:48.973 INFO:teuthology.orchestra.run.smithi159.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-02T12:18:48.973 INFO:teuthology.orchestra.run.smithi159.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-02T12:18:48.973 INFO:teuthology.orchestra.run.smithi159.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-02T12:18:48.973 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:49.018 INFO:teuthology.orchestra.run.smithi159.stderr: 0K .......... .......... .......... .......... .......... 8% 1.08M 1s 2024-04-02T12:18:49.050 INFO:teuthology.orchestra.run.smithi159.stderr: 50K .......... .......... .......... .......... .......... 16% 1.56M 0s 2024-04-02T12:18:49.050 INFO:teuthology.orchestra.run.smithi159.stderr: 100K .......... .......... .......... .......... .......... 24% 59.7M 0s 2024-04-02T12:18:49.066 INFO:teuthology.orchestra.run.smithi159.stderr: 150K .......... .......... .......... .......... .......... 32% 3.01M 0s 2024-04-02T12:18:49.082 INFO:teuthology.orchestra.run.smithi159.stderr: 200K .......... .......... .......... .......... .......... 41% 3.48M 0s 2024-04-02T12:18:49.083 INFO:teuthology.orchestra.run.smithi159.stderr: 250K .......... .......... .......... .......... .......... 49% 106M 0s 2024-04-02T12:18:49.083 INFO:teuthology.orchestra.run.smithi159.stderr: 300K .......... .......... .......... .......... .......... 57% 78.2M 0s 2024-04-02T12:18:49.098 INFO:teuthology.orchestra.run.smithi159.stderr: 350K .......... .......... .......... .......... .......... 65% 3.09M 0s 2024-04-02T12:18:49.114 INFO:teuthology.orchestra.run.smithi159.stderr: 400K .......... .......... .......... .......... .......... 74% 3.41M 0s 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr: 450K .......... .......... .......... .......... .......... 82% 94.7M 0s 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr: 500K .......... .......... .......... .......... .......... 90% 106M 0s 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr: 550K .......... .......... .......... .......... .......... 98% 78.3M 0s 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr: 600K ...... 100% 11.4T=0.1s 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02 12:18:49 (4.22 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-02T12:18:49.115 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:18:49.153 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/ 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/collectl.conf 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/ 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Socket.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/colmux.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Import.html 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/style.css 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-02T12:18:49.154 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Process.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Export.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-02T12:18:49.155 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Startup.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Performance.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Messages.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Logging.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Misc.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Examples.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-02T12:18:49.156 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Data.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Playback.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Hello.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Network.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Memory.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/Disks.html 2024-04-02T12:18:49.157 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/service/ 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/service/collectl.service 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/initd/ 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/initd/collectl 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/misc.ph 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/gexpr.ph 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/statsd.ph 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/envrules.std 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/hello.ph 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/vmsum.ph 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/proctree.ph 2024-04-02T12:18:49.158 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/README-WINDOWS 2024-04-02T12:18:49.159 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/formatit.ph 2024-04-02T12:18:49.159 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/colmux 2024-04-02T12:18:49.159 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/INSTALL 2024-04-02T12:18:49.159 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/graphite.ph 2024-04-02T12:18:49.160 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-02T12:18:49.160 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/vmstat.ph 2024-04-02T12:18:49.160 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/README 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/man1/ 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/man1/collectl.1 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/man1/colmux.1 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/lexpr.ph 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/client.pl 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/collectl 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/COPYING 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/vnet.ph 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/GPL 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/UNINSTALL 2024-04-02T12:18:49.161 INFO:teuthology.orchestra.run.smithi159.stdout:collectl-4.3.1/ARTISTIC 2024-04-02T12:18:49.209 INFO:teuthology.orchestra.run.smithi159.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-02T12:18:49.210 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-02T12:18:49.795 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:05:16 ago on Tue 02 Apr 2024 12:13:33 PM UTC. 2024-04-02T12:18:49.897 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-02T12:18:49.900 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-02T12:18:49.960 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout: librbd-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 23 k 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout: perf x86_64 5.14.0-434.el9 CentOS-AppStream 2.2 M 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-04-02T12:18:49.961 INFO:teuthology.orchestra.run.smithi159.stdout: libradospp-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 30 k 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout:Install 5 Packages 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 2.3 M 2024-04-02T12:18:49.962 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 11 M 2024-04-02T12:18:49.963 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-04-02T12:18:57.659 INFO:teuthology.orchestra.run.smithi159.stdout:(1/5): libradospp-devel-19.0.0-2589.g38caedd8.e 64 kB/s | 30 kB 00:00 2024-04-02T12:18:57.701 INFO:teuthology.orchestra.run.smithi159.stdout:(2/5): perf-5.14.0-434.el9.x86_64.rpm 4.2 MB/s | 2.2 MB 00:00 2024-04-02T12:18:57.717 INFO:teuthology.orchestra.run.smithi159.stdout:(3/5): librbd-devel-19.0.0-2589.g38caedd8.el9.x 44 kB/s | 23 kB 00:00 2024-04-02T12:18:57.743 INFO:teuthology.orchestra.run.smithi159.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 315 kB/s | 13 kB 00:00 2024-04-02T12:18:57.793 INFO:teuthology.orchestra.run.smithi159.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 904 kB/s | 118 kB 00:00 2024-04-02T12:18:57.795 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:18:57.795 INFO:teuthology.orchestra.run.smithi159.stdout:Total 306 kB/s | 2.3 MB 00:07 2024-04-02T12:18:57.796 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-04-02T12:18:57.816 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-04-02T12:18:57.816 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-04-02T12:18:57.941 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-04-02T12:18:57.941 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-04-02T12:18:58.225 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-04-02T12:18:58.295 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-02T12:18:58.325 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-02T12:18:58.365 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradospp-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 3/5 2024-04-02T12:18:58.676 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librbd-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 4/5 2024-04-02T12:18:58.771 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : perf-5.14.0-434.el9.x86_64 5/5 2024-04-02T12:18:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: perf-5.14.0-434.el9.x86_64 5/5 2024-04-02T12:18:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradospp-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/5 2024-04-02T12:18:59.727 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2/5 2024-04-02T12:18:59.728 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : perf-5.14.0-434.el9.x86_64 3/5 2024-04-02T12:18:59.728 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: libradospp-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: librbd-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: pdsh-2.34-7.el9.x86_64 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: perf-5.14.0-434.el9.x86_64 2024-04-02T12:19:00.062 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-04-02T12:19:00.063 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-04-02T12:19:00.392 INFO:tasks.cbt:benchmark: radosbench 2024-04-02T12:19:00.392 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-02T12:19:00.421 DEBUG:teuthology.orchestra.run.smithi159:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - Settings.cluster: 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: 'clients': ['ubuntu@smithi159.front.sepia.ceph.com'], 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: 'head': 'ubuntu@smithi159.front.sepia.ceph.com', 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: 'is_teuthology': True, 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: 'iterations': 1, 2024-04-02T12:19:00.831 INFO:teuthology.orchestra.run.smithi159.stderr: 'mons': ['ubuntu@smithi159.front.sepia.ceph.com'], 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'osds': ['ubuntu@smithi159.front.sepia.ceph.com'], 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'osds_per_node': 3, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'pid_dir': '/var/run/ceph', 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'pgp_size': 256, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'replication': 'replicated'}}, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'rebuild_every_test': False, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'rgws': {'ubuntu@smithi159.front.sepia.ceph.com': None}, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'use_existing': True, 2024-04-02T12:19:00.832 INFO:teuthology.orchestra.run.smithi159.stderr: 'user': 'ubuntu'} 2024-04-02T12:19:00.851 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-04-02T12:19:00.853 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:19:00.853 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:19:00.853 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-04-02T12:19:00.856 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:19:00.856 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:19:00.858 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:19:00.858 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:19:00.858 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-04-02T12:19:00.861 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:19:00.861 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:19:00.861 INFO:teuthology.orchestra.run.smithi159.stderr:12:19:00 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:20:00.903 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:00.904 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:20:00.937 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:00.937 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:20:00.980 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:00.981 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-04-02T12:20:00.982 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:00.982 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-04-02T12:20:00.984 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:00.984 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:00.984 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-04-02T12:20:01.001 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.001 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/$(basename ${i}).smithi159.front.sepia.ceph.com ; done 2024-04-02T12:20:01.006 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.006 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.006 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-04-02T12:20:01.042 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - WARNING - cbt - sudo killall -9 rados 2024-04-02T12:20:01.042 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - WARNING - cbt - error 1 seen, continuing anyway... 2024-04-02T12:20:01.061 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c. 2024-04-02T12:20:01.061 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-04-02T12:20:01.061 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.061 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.061 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-04-02T12:20:01.077 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.078 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.078 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-04-02T12:20:01.080 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.080 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:20:01.082 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.082 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.082 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-04-02T12:20:01.085 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.085 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:20:01.085 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:20:01.085 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.085 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.085 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-04-02T12:20:01.403 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.403 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:01.403 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-04-02T12:20:02.633 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:02 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:02.634 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:02 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:02.634 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-04-02T12:20:03.637 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:03 - INFO - cbt - Checking Health after pool creation. 2024-04-02T12:20:08.643 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:08 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:08.643 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:08 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:08.643 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-02T12:20:09.239 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:09 - INFO - cbt - Final Pool Health Check. 2024-04-02T12:20:14.244 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.245 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.245 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-02T12:20:14.574 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.574 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:20:14.597 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.598 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:20:14.636 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.636 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-04-02T12:20:14.637 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.638 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-04-02T12:20:14.639 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.639 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.639 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-02T12:20:14.760 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:14.760 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-02T12:20:15.326 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:15.326 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:15.326 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-02T12:20:15.643 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:15.644 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:15.644 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-04-02T12:20:15.685 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:15.685 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:15.685 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-04-02T12:20:16.432 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.432 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.432 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-04-02T12:20:16.433 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.433 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:20:16.435 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.435 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.435 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-04-02T12:20:16.438 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.438 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - INFO - cbt - Running radosbench write test. 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:20:16.439 INFO:teuthology.orchestra.run.smithi159.stderr:12:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-04-02T12:21:17.607 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:17.607 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:21:17.671 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:17.671 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:21:17.712 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:17.712 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-04-02T12:21:17.714 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:17.714 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-04-02T12:21:17.715 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:17.715 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:17.715 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-04-02T12:21:18.022 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.030 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.030 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-04-02T12:21:18.047 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.047 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/$(basename ${i}).smithi159.front.sepia.ceph.com ; done 2024-04-02T12:21:18.081 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - INFO - cbt - Convert results to json format. 2024-04-02T12:21:18.081 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.082 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.082 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.082 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-02T12:21:18.391 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.391 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-02T12:21:18.716 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.716 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:18.716 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-02T12:21:19.063 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.063 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.063 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand 2024-04-02T12:21:19.126 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.126 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.126 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/ceph_settings.out 2024-04-02T12:21:19.886 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.887 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.887 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl 2024-04-02T12:21:19.889 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.889 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:21:19.890 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.891 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.891 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf 2024-04-02T12:21:19.894 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.894 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:21:19.894 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:21:19.894 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - INFO - cbt - Running radosbench rand test. 2024-04-02T12:21:19.894 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.894 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.895 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.0 2024-04-02T12:21:19.895 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.895 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:21:19.896 INFO:teuthology.orchestra.run.smithi159.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.1 2024-04-02T12:22:21.060 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.060 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:22:21.099 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.099 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:22:21.136 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.136 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf.data 2024-04-02T12:22:21.138 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.138 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.* 2024-04-02T12:22:21.140 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.140 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.140 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/historic_ops.out 2024-04-02T12:22:21.479 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.490 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.490 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* 2024-04-02T12:22:21.507 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - get_fqdn_local()=smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.507 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/$(basename ${i}).smithi159.front.sepia.ceph.com ; done 2024-04-02T12:22:21.563 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - INFO - cbt - Convert results to json format. 2024-04-02T12:22:21.563 INFO:teuthology.orchestra.run.smithi159.stderr:12:22:21 - DEBUG - cbt - Nodes : ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:21.832 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-02T12:22:21.841 DEBUG:teuthology.orchestra.run.smithi159:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-02T12:22:23.342 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-02T12:22:25.015 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-02T12:22:25.016 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-02T12:22:25.077 INFO:teuthology.orchestra.run.smithi159.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi159.front.sepia.ceph.com 2024-04-02T12:22:25.077 INFO:teuthology.orchestra.run.smithi159.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi159.front.sepia.ceph.com 2024-04-02T12:22:25.077 INFO:teuthology.orchestra.run.smithi159.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi159.front.sepia.ceph.com 2024-04-02T12:22:25.077 INFO:teuthology.orchestra.run.smithi159.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi159.front.sepia.ceph.com 2024-04-02T12:22:25.078 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:22:25.078 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi159.front.sepia.ceph.com of=/dev/stdout 2024-04-02T12:22:25.143 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T12:22:25.143 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2024-04-02T12:22:25.202 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T12:22:25.202 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T12:22:25.202 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:22:25.202 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi159.front.sepia.ceph.com of=/dev/stdout 2024-04-02T12:22:25.261 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T12:22:25.261 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2024-04-02T12:22:25.319 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T12:22:25.319 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T12:22:25.319 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:22:25.319 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi159.front.sepia.ceph.com of=/dev/stdout 2024-04-02T12:22:25.379 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T12:22:25.380 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2024-04-02T12:22:25.438 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T12:22:25.438 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T12:22:25.439 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-04-02T12:22:25.439 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi159.front.sepia.ceph.com of=/dev/stdout 2024-04-02T12:22:25.501 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T12:22:25.502 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2024-04-02T12:22:25.559 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T12:22:25.560 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T12:22:25.576 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T12:22:25.584 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T12:22:25.596 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T12:22:25.604 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T12:22:25.604 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-02T12:22:25.860 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-02T12:22:25.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:22:25.997+0000 7fc315e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:22:25.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-04-02T12:22:26.006+0000 7fc315e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:22:26.205 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-02T12:22:26.205 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-02T12:22:26.205 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-02T12:22:26.205 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:22:30.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-02T12:22:32.307 INFO:tasks.ceph.osd.0:Stopped 2024-04-02T12:22:32.308 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-02T12:22:32.308 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:22:36.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-02T12:22:36.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-02T12:22:38.409 INFO:tasks.ceph.osd.1:Stopped 2024-04-02T12:22:38.535 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-02T12:22:38.535 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:22:41.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-02T12:22:41.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-02T12:22:44.636 INFO:tasks.ceph.osd.2:Stopped 2024-04-02T12:22:44.637 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-02T12:22:44.637 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-02T12:22:44.637 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:22:44.671 INFO:tasks.ceph.mgr.x:Stopped 2024-04-02T12:22:44.671 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-02T12:22:44.671 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-02T12:22:44.671 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:22:44.724 INFO:tasks.ceph.mon.a:Stopped 2024-04-02T12:22:44.724 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-02T12:22:44.725 DEBUG:teuthology.orchestra.run.smithi159:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-02T12:22:44.828 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:44.828 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-02T12:22:45.029 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:45.030 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-02T12:22:45.109 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:45.110 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-02T12:22:45.168 INFO:tasks.ceph:Archiving mon data... 2024-04-02T12:22:45.169 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514/data/mon.a.tgz 2024-04-02T12:22:45.169 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-04-02T12:22:45.188 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.XHO8Wmr5Zx 2024-04-02T12:22:45.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.XHO8Wmr5Zx 2024-04-02T12:22:45.319 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.XHO8Wmr5Zx 2024-04-02T12:22:45.431 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.XHO8Wmr5Zx is 193KB 2024-04-02T12:22:45.484 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.XHO8Wmr5Zx 2024-04-02T12:22:45.500 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-02T12:22:45.500 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T12:22:45.622 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-02T12:22:45.622 INFO:tasks.ceph:Archiving crash dumps... 2024-04-02T12:22:45.624 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514/remote/smithi159/crash 2024-04-02T12:22:45.625 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T12:22:45.688 INFO:tasks.ceph:Compressing logs... 2024-04-02T12:22:45.688 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T12:22:45.804 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31135.log 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31386.log 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31135.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31363.log 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.33243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31472.log 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz 2024-04-02T12:22:45.805 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32466.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31574.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30546.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30358.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32668.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30080.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32668.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32667.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31177.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32667.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30951.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31366.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.30951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30318.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31258.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30278.log 2024-04-02T12:22:45.806 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32942.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31093.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32942.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr: 65.3% -- replaced with /var/log/ceph/ceph-client.admin.32942.log.gz 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31007.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28709.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30168.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.tmp-client.admin.28709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32603.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30168.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32943.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31532.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32943.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr: 65.3% -- replaced with /var/log/ceph/ceph-client.admin.32943.log.gz 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31532.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32580.log 2024-04-02T12:22:45.807 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31049.log 2024-04-02T12:22:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31259.log 2024-04-02T12:22:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32557.log 2024-04-02T12:22:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.31259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32530.log 2024-04-02T12:22:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30516.log 2024-04-02T12:22:45.808 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.32530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31260.log 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.30516.log: /var/log/ceph/ceph-client.admin.31260.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31135.log.gz 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31177.log.gz 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31363.log.gz 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31258.log.gz 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31472.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31386.log.gz 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31366.log.gz 2024-04-02T12:22:45.812 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31574.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31532.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30168.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30080.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.32668.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 86.2% 65.6% -- replaced with /var/log/ceph/ceph.log.gz -- replaced with /var/log/ceph/ceph-client.admin.32667.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31260.log.gz 2024-04-02T12:22:45.813 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31259.log.gz 2024-04-02T12:22:45.820 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30546.log.gz 2024-04-02T12:22:45.823 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30278.log.gz 2024-04-02T12:22:45.823 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30318.log.gz 2024-04-02T12:22:45.824 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30358.log.gz 2024-04-02T12:22:45.829 INFO:teuthology.orchestra.run.smithi159.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-02T12:22:45.835 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30516.log.gz 2024-04-02T12:22:45.860 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32466.log.gz 2024-04-02T12:22:45.860 INFO:teuthology.orchestra.run.smithi159.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.32530.log.gz 2024-04-02T12:22:45.861 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32557.log.gz 2024-04-02T12:22:45.861 INFO:teuthology.orchestra.run.smithi159.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.32580.log.gz 2024-04-02T12:22:45.861 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32603.log.gz 2024-04-02T12:22:45.861 INFO:teuthology.orchestra.run.smithi159.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-02T12:22:45.869 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30951.log.gz 2024-04-02T12:22:45.869 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31007.log.gz 2024-04-02T12:22:45.869 INFO:teuthology.orchestra.run.smithi159.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31049.log.gz 2024-04-02T12:22:45.869 INFO:teuthology.orchestra.run.smithi159.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31093.log.gz 2024-04-02T12:22:45.957 INFO:teuthology.orchestra.run.smithi159.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-02T12:22:45.989 INFO:teuthology.orchestra.run.smithi159.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28709.log.gz 2024-04-02T12:22:46.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-02T12:22:55.572 INFO:teuthology.orchestra.run.smithi159.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-02T12:22:55.907 INFO:teuthology.orchestra.run.smithi159.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-02T12:22:56.765 INFO:teuthology.orchestra.run.smithi159.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-02T12:22:56.767 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-04-02T12:22:56.767 INFO:teuthology.orchestra.run.smithi159.stderr:real 0m11.025s 2024-04-02T12:22:56.767 INFO:teuthology.orchestra.run.smithi159.stderr:user 0m8.063s 2024-04-02T12:22:56.767 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m0.418s 2024-04-02T12:22:56.767 INFO:tasks.ceph:Archiving logs... 2024-04-02T12:22:56.767 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514/remote/smithi159/log 2024-04-02T12:22:56.768 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T12:22:58.007 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T12:22:58.017 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T12:22:58.017 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T12:22:58.075 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T12:22:58.083 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T12:22:58.083 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T12:22:58.103 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-04-02T12:22:58.158 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T12:22:58.159 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-04-02T12:22:58.159 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +355us[ +355us] +/- 97ms 2024-04-02T12:22:58.159 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -767us[ -769us] +/- 51ms 2024-04-02T12:22:58.159 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +1001us[+1001us] +/- 95ms 2024-04-02T12:22:58.159 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:22:58.174 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T12:22:58.183 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T12:22:58.183 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T12:22:58.192 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T12:22:58.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T12:22:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:22:58.357 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T12:22:58.367 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T12:22:58.376 INFO:teuthology.task.internal:Duration was 835.937858 seconds 2024-04-02T12:22:58.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T12:22:58.385 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T12:22:58.385 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T12:22:58.538 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T12:22:58.668 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T12:22:58.668 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-04-02T12:22:58.668 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T12:22:58.687 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T12:22:58.687 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T12:22:58.756 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T12:22:58.757 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T12:22:58.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T12:22:58.925 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T12:22:58.925 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T12:22:58.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T12:22:58.973 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T12:22:59.047 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-04-02T12:22:59.065 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T12:22:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:22:59.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T12:22:59.089 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T12:22:59.089 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635514/remote/smithi159 2024-04-02T12:22:59.089 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T12:22:59.221 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T12:22:59.222 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T12:22:59.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T12:22:59.281 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T12:22:59.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T12:22:59.290 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T12:22:59.290 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T12:22:59.325 INFO:teuthology.orchestra.run.smithi159.stdout: 265819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 12:22 /home/ubuntu/cephtest 2024-04-02T12:22:59.326 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T12:22:59.336 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T12:22:59.360 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-testing-apr-2&var-job_id_selected=7635514 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} duration: 835.9378581047058 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-04-02T12:22:59.361 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T12:22:59.454 INFO:teuthology.run:pass