2024-04-02T12:16:58.118 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T12:16:58.118 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} --archive /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527 --name matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi --owner scheduled_matan@teuthology -- /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527/orig.config.yaml 2024-04-02T12:16:58.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T12:16:58.258 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527 branch: wip-matanb-crimson-testing-apr-2 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} email: null first_in_suite: false job_id: '7635527' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-apr-2 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 107e1624f2a9b72a56ee529969558944d24a2a75 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 223 sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 107e1624f2a9b72a56ee529969558944d24a2a75 targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7cKe3M/hlEtYGKZhQBrbfmzpghFhhXH2QRAGfr8SGEZFK+46HggvnvpGb+1Xq0cwXjhWfCwMsCX7CukabtRAI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_11:03:04 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T12:16:58.258 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa; will attempt to use it 2024-04-02T12:16:58.259 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa/tasks 2024-04-02T12:16:58.259 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T12:16:58.261 INFO:teuthology.task.internal:Checking packages... 2024-04-02T12:16:58.316 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '38caedd896c6b19ed5e43655372d95dc8e78cfda' 2024-04-02T12:16:58.316 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T12:16:58.316 INFO:teuthology.packaging:ref: None 2024-04-02T12:16:58.316 INFO:teuthology.packaging:tag: None 2024-04-02T12:16:58.316 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-apr-2 2024-04-02T12:16:58.316 INFO:teuthology.packaging:sha1: 38caedd896c6b19ed5e43655372d95dc8e78cfda 2024-04-02T12:16:58.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-apr-2 2024-04-02T12:16:58.536 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2589.g38caedd8 2024-04-02T12:16:58.538 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T12:16:58.545 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T12:16:58.545 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T12:16:58.552 INFO:teuthology.task.internal:Saving configuration 2024-04-02T12:16:58.568 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T12:16:58.574 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T12:16:58.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 12:11:59.699607', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7cKe3M/hlEtYGKZhQBrbfmzpghFhhXH2QRAGfr8SGEZFK+46HggvnvpGb+1Xq0cwXjhWfCwMsCX7CukabtRAI='} 2024-04-02T12:16:58.600 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T12:16:58.607 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-02T12:16:58.607 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T12:16:58.652 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6ae249ef70>, signals=[15]) 2024-04-02T12:16:58.652 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T12:16:58.658 INFO:teuthology.task.internal:Opening connections... 2024-04-02T12:16:58.658 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:16:58.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:16:58.752 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T12:16:58.759 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-04-02T12:16:58.827 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-04-02T12:16:58.828 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-04-02T12:16:58.883 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-04-02T12:16:58.883 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T12:16:58.884 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T12:16:58.885 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T12:16:58.885 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-04-02T12:16:58.914 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T12:16:58.923 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T12:16:58.928 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T12:16:58.929 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-04-02T12:16:58.944 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T12:16:58.949 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T12:16:58.949 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-04-02T12:16:59.014 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T12:16:59.048 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T12:16:59.081 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-02T12:16:59.081 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-02T12:16:59.081 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T12:16:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:16:59.097 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-04-02T12:16:59.154 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-04-02T12:16:59.155 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-04-02T12:16:59.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-04-02T12:17:01.235 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-02T12:17:02.952 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-04-02T12:17:04.384 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 7.3 MB/s | 8.0 MB 00:01 2024-04-02T12:17:08.718 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-04-02T12:17:20.576 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 753 kB/s | 6.1 MB 00:08 2024-04-02T12:17:22.337 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-04-02T12:17:24.323 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-02T12:17:29.262 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-02T12:17:31.544 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T12:17:31.544 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T12:17:31.635 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:17:31.644 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:17:31.644 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-04-02T12:17:31.645 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:17:31.646 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 76 M 2024-04-02T12:17:31.646 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-04-02T12:17:31.646 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:17:31.830 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-04-02T12:17:32.480 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-04-02T12:17:33.022 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-04-02T12:17:33.464 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-02T12:17:33.466 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:17:33.467 INFO:teuthology.orchestra.run.smithi129.stdout:Total 42 MB/s | 76 MB 00:01 2024-04-02T12:17:33.467 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:17:33.854 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:17:33.855 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:17:34.691 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:17:34.691 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:17:37.001 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:17:37.678 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T12:17:37.764 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T12:17:40.507 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T12:17:40.648 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T12:17:45.380 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T12:17:45.421 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:17:50.185 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:18:19.589 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:18:19.589 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T12:18:19.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:18:19.632 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:18:20.015 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:18:20.015 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T12:18:20.015 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T12:18:20.015 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:18:20.435 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:18:20.804 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T12:18:20.921 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T12:18:20.923 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T12:18:20.923 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T12:18:20.923 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T12:18:20.923 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T12:18:20.923 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-02T12:18:20.923 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T12:18:20.923 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T12:18:21.002 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T12:18:21.003 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T12:18:21.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-04-02T12:18:21.045 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-04-02T12:18:21.046 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-04-02T12:18:21.046 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T12:18:21.046 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T12:18:21.046 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T12:18:21.046 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:18:21.047 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T12:18:21.048 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T12:18:21.048 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-434.el9.x86_64 2024-04-02T12:18:21.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T12:18:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-04-02T12:18:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T12:18:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-04-02T12:18:23.192 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-04-02T12:18:23.208 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.WNLulx6geR 2024-04-02T12:18:23.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WNLulx6geR 2024-04-02T12:18:23.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.WNLulx6geR 2024-04-02T12:18:23.413 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.WNLulx6geR is 6KB 2024-04-02T12:18:23.462 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.WNLulx6geR 2024-04-02T12:18:23.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T12:18:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T12:18:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T12:18:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T12:18:23.541 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T12:18:23.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T12:18:23.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-04-02T12:18:53.718 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-02T12:18:53.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-02T12:18:53.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:19:12.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-04-02T12:19:21.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-02T12:19:21.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:19:24.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-04-02T12:19:36.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-02T12:19:36.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:20:08.159 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-04-02T12:20:08.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-02T12:20:08.837 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-02T12:20:08.837 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-04-02T12:20:08.850 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-434.el9.x86_64 2024-04-02T12:20:08.851 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T12:20:08.851 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T12:20:08.851 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T12:20:09.851 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-02T12:20:09.852 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T12:20:09.973 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-04-02T12:20:10.007 DEBUG:teuthology.parallel:result is None 2024-04-02T12:20:10.008 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T12:20:10.016 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T12:20:10.016 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T12:20:10.042 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T12:20:10.049 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T12:20:10.056 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T12:20:10.057 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T12:20:10.122 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T12:20:10.130 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T12:20:10.130 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T12:20:10.220 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T12:20:10.240 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T12:20:10.242 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T12:20:10.252 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T12:20:10.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T12:20:10.297 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T12:20:10.307 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T12:20:10.307 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T12:20:10.369 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T12:20:10.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T12:20:10.534 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T12:20:10.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T12:20:10.665 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:20:10.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T12:20:10.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-04-02T12:20:10.958 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T12:20:11.511 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T12:20:11.519 INFO:teuthology.task.internal:Starting timer... 2024-04-02T12:20:11.519 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T12:20:11.574 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T12:20:11.582 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-02T12:20:11.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-04-02T12:20:11.638 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-04-02T12:20:11.640 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T12:20:11.640 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-04-02T12:20:11.666 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-04-02T12:20:11.666 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-04-02T12:20:11.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T12:20:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:20:11.737 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T12:20:11.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-04-02T12:20:11.818 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T12:20:11.828 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-02T12:20:11.833 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-02T12:20:11.948 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T12:20:11.964 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T12:20:11.965 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T12:24:30.587 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-04-02T12:24:30.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-02T12:24:30.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T12:24:30.702 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-04-02T12:24:30.777 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-02T12:24:30.778 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T12:24:30.788 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T12:24:30.788 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T12:24:30.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T12:24:30.850 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T12:24:30.866 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T12:24:30.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-04-02T12:24:30.909 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-04-02T12:24:30.926 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T12:24:30.943 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T12:24:30.994 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-04-02T12:24:30.997 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T12:24:30.997 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-04-02T12:24:30.997 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:24:30.997 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:24:30.997 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:24:30.998 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:24:30.999 INFO:teuthology.run_tasks:Running task install... 2024-04-02T12:24:31.007 DEBUG:teuthology.task.install:project ceph 2024-04-02T12:24:31.007 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'}} 2024-04-02T12:24:31.007 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'} 2024-04-02T12:24:31.007 INFO:teuthology.task.install:Using flavor: crimson 2024-04-02T12:24:31.012 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T12:24:31.013 INFO:teuthology.task.install:extra packages: [] 2024-04-02T12:24:31.013 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda', 'tag': None, 'wait_for_package': False} 2024-04-02T12:24:31.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=38caedd896c6b19ed5e43655372d95dc8e78cfda 2024-04-02T12:24:31.310 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/ 2024-04-02T12:24:31.310 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2589.g38caedd8 2024-04-02T12:24:31.493 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-apr-2/38caedd896c6b19ed5e43655372d95dc8e78cfda/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T12:24:31.493 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:24:31.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T12:24:31.521 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-02T12:24:31.521 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/38caedd896c6b19ed5e43655372d95dc8e78cfda/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T12:24:31.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T12:24:31.670 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T12:24:31.736 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-04-02T12:24:31.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-04-02T12:24:31.981 INFO:teuthology.orchestra.run.smithi129.stdout:74 files removed 2024-04-02T12:24:32.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2024-04-02T12:24:32.636 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 267 kB/s | 87 kB 00:00 2024-04-02T12:24:33.070 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-02T12:24:33.404 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 6.9 kB/s | 2.0 kB 00:00 2024-04-02T12:24:34.014 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T12:24:35.548 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-04-02T12:24:36.505 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-02T12:24:40.533 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 6.4 MB/s | 19 MB 00:02 2024-04-02T12:24:44.559 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-04-02T12:26:55.271 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 123 B/s | 16 kB 02:09 2024-04-02T12:26:57.023 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-02T12:27:02.104 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 7.0 kB/s | 1.7 kB 00:00 2024-04-02T12:27:04.746 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 9.6 M 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 3.4 M 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 3.7 M 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T12:27:04.758 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 5.9 M 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 19 M 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 25 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 795 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 548 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 5.0 M 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T12:27:04.759 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 45 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 134 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 159 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 315 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 294 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 99 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T12:27:04.760 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout:Install 30 Packages 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:04.761 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 58 M 2024-04-02T12:27:04.762 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:27:08.114 INFO:teuthology.orchestra.run.smithi129.stdout:(1/32): ceph-base-19.0.0-2589.g38caedd8.el9.x86 1.9 MB/s | 5.9 MB 00:03 2024-04-02T12:27:08.164 INFO:teuthology.orchestra.run.smithi129.stdout:(2/32): ceph-selinux-19.0.0-2589.g38caedd8.el9. 505 kB/s | 25 kB 00:00 2024-04-02T12:27:08.364 INFO:teuthology.orchestra.run.smithi129.stdout:(3/32): libcephfs2-19.0.0-2589.g38caedd8.el9.x8 3.9 MB/s | 795 kB 00:00 2024-04-02T12:27:08.498 INFO:teuthology.orchestra.run.smithi129.stdout:(4/32): libradosstriper1-19.0.0-2589.g38caedd8. 4.0 MB/s | 548 kB 00:00 2024-04-02T12:27:08.956 INFO:teuthology.orchestra.run.smithi129.stdout:(5/32): ceph-radosgw-19.0.0-2589.g38caedd8.el9. 2.4 MB/s | 9.6 MB 00:04 2024-04-02T12:27:09.007 INFO:teuthology.orchestra.run.smithi129.stdout:(6/32): python3-ceph-argparse-19.0.0-2589.g38ca 909 kB/s | 45 kB 00:00 2024-04-02T12:27:09.073 INFO:teuthology.orchestra.run.smithi129.stdout:(7/32): python3-ceph-common-19.0.0-2589.g38caed 2.0 MB/s | 134 kB 00:00 2024-04-02T12:27:09.140 INFO:teuthology.orchestra.run.smithi129.stdout:(8/32): python3-cephfs-19.0.0-2589.g38caedd8.el 2.3 MB/s | 159 kB 00:00 2024-04-02T12:27:09.232 INFO:teuthology.orchestra.run.smithi129.stdout:(9/32): python3-rados-19.0.0-2589.g38caedd8.el9 3.4 MB/s | 315 kB 00:00 2024-04-02T12:27:09.358 INFO:teuthology.orchestra.run.smithi129.stdout:(10/32): python3-rbd-19.0.0-2589.g38caedd8.el9. 2.3 MB/s | 294 kB 00:00 2024-04-02T12:27:09.408 INFO:teuthology.orchestra.run.smithi129.stdout:(11/32): python3-rgw-19.0.0-2589.g38caedd8.el9. 1.9 MB/s | 99 kB 00:00 2024-04-02T12:27:09.550 INFO:teuthology.orchestra.run.smithi129.stdout:(12/32): librgw2-19.0.0-2589.g38caedd8.el9.x86_ 4.8 MB/s | 5.0 MB 00:01 2024-04-02T12:27:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-04-02T12:27:09.667 INFO:teuthology.orchestra.run.smithi129.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-04-02T12:27:09.692 INFO:teuthology.orchestra.run.smithi129.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 482 kB/s | 44 kB 00:00 2024-04-02T12:27:09.851 INFO:teuthology.orchestra.run.smithi129.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-04-02T12:27:09.910 INFO:teuthology.orchestra.run.smithi129.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 282 kB 00:00 2024-04-02T12:27:09.934 INFO:teuthology.orchestra.run.smithi129.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 494 kB/s | 41 kB 00:00 2024-04-02T12:27:10.123 INFO:teuthology.orchestra.run.smithi129.stdout:(19/32): ceph-common-19.0.0-2589.g38caedd8.el9. 3.6 MB/s | 19 MB 00:05 2024-04-02T12:27:10.169 INFO:teuthology.orchestra.run.smithi129.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 849 kB/s | 220 kB 00:00 2024-04-02T12:27:10.185 INFO:teuthology.orchestra.run.smithi129.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 139 kB/s | 35 kB 00:00 2024-04-02T12:27:10.202 INFO:teuthology.orchestra.run.smithi129.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 559 kB/s | 46 kB 00:00 2024-04-02T12:27:10.544 INFO:teuthology.orchestra.run.smithi129.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 76 kB/s | 26 kB 00:00 2024-04-02T12:27:10.686 INFO:teuthology.orchestra.run.smithi129.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 617 kB/s | 309 kB 00:00 2024-04-02T12:27:10.703 INFO:teuthology.orchestra.run.smithi129.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-04-02T12:27:10.770 INFO:teuthology.orchestra.run.smithi129.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-04-02T12:27:10.887 INFO:teuthology.orchestra.run.smithi129.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-02T12:27:11.029 INFO:teuthology.orchestra.run.smithi129.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-04-02T12:27:11.063 INFO:teuthology.orchestra.run.smithi129.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-04-02T12:27:11.180 INFO:teuthology.orchestra.run.smithi129.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.4 MB/s | 1.6 MB 00:00 2024-04-02T12:27:11.613 INFO:teuthology.orchestra.run.smithi129.stdout:(31/32): librados2-19.0.0-2589.g38caedd8.el9.x8 5.8 MB/s | 3.4 MB 00:00 2024-04-02T12:27:11.672 INFO:teuthology.orchestra.run.smithi129.stdout:(32/32): librbd1-19.0.0-2589.g38caedd8.el9.x86_ 6.1 MB/s | 3.7 MB 00:00 2024-04-02T12:27:11.678 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:27:11.678 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.4 MB/s | 58 MB 00:06 2024-04-02T12:27:11.679 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:27:11.716 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:27:11.716 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:27:12.198 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:27:12.198 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:27:13.682 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:27:14.141 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-02T12:27:14.190 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 2/34 2024-04-02T12:27:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 2/34 2024-04-02T12:27:14.387 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 3/34 2024-04-02T12:27:14.425 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 4/34 2024-04-02T12:27:14.479 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 4/34 2024-04-02T12:27:14.534 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-02T12:27:14.563 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-02T12:27:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-02T12:27:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-02T12:27:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-02T12:27:14.820 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-02T12:27:14.852 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-02T12:27:15.204 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-02T12:27:15.232 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 11/34 2024-04-02T12:27:15.298 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 11/34 2024-04-02T12:27:15.341 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-2589.g38caedd8.el9. 12/34 2024-04-02T12:27:15.413 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 13/34 2024-04-02T12:27:15.445 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 14/34 2024-04-02T12:27:15.523 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-02T12:27:15.559 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_6 16/34 2024-04-02T12:27:15.639 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_6 16/34 2024-04-02T12:27:15.675 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-02T12:27:15.782 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-02T12:27:15.809 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-02T12:27:15.857 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-02T12:27:15.891 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-02T12:27:15.920 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-02T12:27:15.951 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-02T12:27:15.992 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-02T12:27:16.526 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-02T12:27:16.557 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 26/34 2024-04-02T12:27:16.637 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 26/34 2024-04-02T12:27:16.714 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 27/34 2024-04-02T12:27:16.751 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-2589.g38caedd8.el9.x8 28/34 2024-04-02T12:27:19.073 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 29/34 2024-04-02T12:27:19.106 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 29/34 2024-04-02T12:27:19.799 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 29/34 2024-04-02T12:27:19.842 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 30/34 2024-04-02T12:27:19.934 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 30/34 2024-04-02T12:27:19.934 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T12:27:19.934 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T12:27:19.934 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:19.969 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 31/34 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 31/34 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-04-02T12:27:29.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:30.888 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 32/34 2024-04-02T12:27:30.912 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 32/34 2024-04-02T12:27:30.912 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:27:30.912 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T12:27:30.912 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T12:27:30.912 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T12:27:30.912 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:30.914 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-02T12:27:30.999 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-02T12:27:30.999 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T12:27:32.270 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T12:27:32.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 1/34 2024-04-02T12:27:32.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 2/34 2024-04-02T12:27:32.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 3/34 2024-04-02T12:27:32.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 4/34 2024-04-02T12:27:32.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 5/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_6 6/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 7/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2589.g38caedd8.el9. 8/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-2589.g38caedd8.el9.x8 9/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 10/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 11/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 12/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 13/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-02T12:27:32.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-02T12:27:32.272 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-02T12:27:32.273 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 31/34 2024-04-02T12:27:32.273 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-02T12:27:32.273 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 33/34 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.339 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.340 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:33.341 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:27:33.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2024-04-02T12:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:32 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:27:34.478 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:27:34.479 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:27:34.479 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:27:34.479 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:27:34.479 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:27:34.479 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 87 M 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 173 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:34.480 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:27:34.481 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:27:34.481 INFO:teuthology.orchestra.run.smithi129.stdout:Install 11 Packages 2024-04-02T12:27:34.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:27:34.481 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 90 M 2024-04-02T12:27:34.481 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 820 M 2024-04-02T12:27:34.481 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:27:35.036 INFO:teuthology.orchestra.run.smithi129.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-04-02T12:27:35.103 INFO:teuthology.orchestra.run.smithi129.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-02T12:27:35.145 INFO:teuthology.orchestra.run.smithi129.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-02T12:27:35.187 INFO:teuthology.orchestra.run.smithi129.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-02T12:27:35.212 INFO:teuthology.orchestra.run.smithi129.stdout:(5/11): libcephsqlite-19.0.0-2589.g38caedd8.el9 545 kB/s | 173 kB 00:00 2024-04-02T12:27:35.238 INFO:teuthology.orchestra.run.smithi129.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 90 kB 00:00 2024-04-02T12:27:35.371 INFO:teuthology.orchestra.run.smithi129.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-02T12:27:35.471 INFO:teuthology.orchestra.run.smithi129.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.0 MB/s | 469 kB 00:00 2024-04-02T12:27:35.572 INFO:teuthology.orchestra.run.smithi129.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 933 kB/s | 187 kB 00:00 2024-04-02T12:27:35.622 INFO:teuthology.orchestra.run.smithi129.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 810 kB/s | 122 kB 00:00 2024-04-02T12:27:45.689 INFO:teuthology.orchestra.run.smithi129.stdout:(11/11): ceph-test-19.0.0-2589.g38caedd8.el9.x8 8.1 MB/s | 87 MB 00:10 2024-04-02T12:27:45.692 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:27:45.692 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.0 MB/s | 90 MB 00:11 2024-04-02T12:27:45.733 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:27:45.757 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:27:45.757 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:27:46.387 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:27:46.387 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:27:47.407 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:27:47.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-02T12:27:47.524 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-02T12:27:47.557 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-02T12:27:47.629 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-02T12:27:47.674 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-02T12:27:47.725 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-02T12:27:47.770 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-02T12:27:47.805 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-02T12:27:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-02T12:27:47.896 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-02T12:27:47.920 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 10/11 2024-04-02T12:27:59.375 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 10/11 2024-04-02T12:27:59.452 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 11/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 11/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 1/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 2/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-02T12:28:05.694 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-02T12:28:05.695 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-02T12:28:05.695 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-02T12:28:05.695 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-02T12:28:05.695 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-02T12:28:06.111 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-02T12:28:06.111 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:06.111 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:28:06.111 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:06.111 INFO:teuthology.orchestra.run.smithi129.stdout: jq-1.6-16.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:06.112 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:28:06.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2024-04-02T12:28:06.970 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:04 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:28:07.123 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 6.5 k 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 2.2 M 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 1.6 M 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 249 k 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 5.0 M 2024-04-02T12:28:07.126 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 71 M 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T12:28:07.127 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T12:28:07.128 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:Install 36 Packages 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 84 M 2024-04-02T12:28:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 922 M 2024-04-02T12:28:07.130 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:28:07.847 INFO:teuthology.orchestra.run.smithi129.stdout:(1/36): ceph-19.0.0-2589.g38caedd8.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2024-04-02T12:28:09.281 INFO:teuthology.orchestra.run.smithi129.stdout:(2/36): ceph-mds-19.0.0-2589.g38caedd8.el9.x86_ 1.3 MB/s | 2.2 MB 00:01 2024-04-02T12:28:09.347 INFO:teuthology.orchestra.run.smithi129.stdout:(3/36): ceph-mgr-19.0.0-2589.g38caedd8.el9.x86_ 907 kB/s | 1.6 MB 00:01 2024-04-02T12:28:09.640 INFO:teuthology.orchestra.run.smithi129.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2589.g38ca 851 kB/s | 249 kB 00:00 2024-04-02T12:28:09.740 INFO:teuthology.orchestra.run.smithi129.stdout:(5/36): ceph-mon-19.0.0-2589.g38caedd8.el9.x86_ 2.7 MB/s | 5.0 MB 00:01 2024-04-02T12:28:09.815 INFO:teuthology.orchestra.run.smithi129.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-04-02T12:28:09.841 INFO:teuthology.orchestra.run.smithi129.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2024-04-02T12:28:09.857 INFO:teuthology.orchestra.run.smithi129.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-04-02T12:28:09.874 INFO:teuthology.orchestra.run.smithi129.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-02T12:28:09.891 INFO:teuthology.orchestra.run.smithi129.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-02T12:28:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-04-02T12:28:09.967 INFO:teuthology.orchestra.run.smithi129.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-02T12:28:09.992 INFO:teuthology.orchestra.run.smithi129.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-04-02T12:28:10.018 INFO:teuthology.orchestra.run.smithi129.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-02T12:28:10.043 INFO:teuthology.orchestra.run.smithi129.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-02T12:28:10.144 INFO:teuthology.orchestra.run.smithi129.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-04-02T12:28:10.177 INFO:teuthology.orchestra.run.smithi129.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 787 kB/s | 125 kB 00:00 2024-04-02T12:28:10.352 INFO:teuthology.orchestra.run.smithi129.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 725 kB/s | 151 kB 00:00 2024-04-02T12:28:10.403 INFO:teuthology.orchestra.run.smithi129.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 216 kB/s | 43 kB 00:00 2024-04-02T12:28:10.428 INFO:teuthology.orchestra.run.smithi129.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-02T12:28:10.478 INFO:teuthology.orchestra.run.smithi129.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-04-02T12:28:10.504 INFO:teuthology.orchestra.run.smithi129.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-04-02T12:28:10.529 INFO:teuthology.orchestra.run.smithi129.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-04-02T12:28:10.554 INFO:teuthology.orchestra.run.smithi129.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-04-02T12:28:10.580 INFO:teuthology.orchestra.run.smithi129.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-04-02T12:28:10.605 INFO:teuthology.orchestra.run.smithi129.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-04-02T12:28:10.630 INFO:teuthology.orchestra.run.smithi129.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-04-02T12:28:10.655 INFO:teuthology.orchestra.run.smithi129.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-02T12:28:10.681 INFO:teuthology.orchestra.run.smithi129.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-02T12:28:10.715 INFO:teuthology.orchestra.run.smithi129.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-04-02T12:28:10.740 INFO:teuthology.orchestra.run.smithi129.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-04-02T12:28:10.800 INFO:teuthology.orchestra.run.smithi129.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 419 kB/s | 36 kB 00:00 2024-04-02T12:28:10.874 INFO:teuthology.orchestra.run.smithi129.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-02T12:28:10.908 INFO:teuthology.orchestra.run.smithi129.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-04-02T12:28:10.933 INFO:teuthology.orchestra.run.smithi129.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 345 kB/s | 20 kB 00:00 2024-04-02T12:28:30.499 INFO:teuthology.orchestra.run.smithi129.stdout:(36/36): ceph-osd-19.0.0-2589.g38caedd8.el9.x86 3.3 MB/s | 71 MB 00:21 2024-04-02T12:28:30.504 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:28:30.504 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.6 MB/s | 84 MB 00:23 2024-04-02T12:28:30.528 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:28:30.572 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:28:30.572 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:28:31.305 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:28:31.306 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:28:32.953 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:28:33.009 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-02T12:28:33.050 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-02T12:28:33.086 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-02T12:28:33.156 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-02T12:28:33.193 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-02T12:28:33.226 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-02T12:28:33.318 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-02T12:28:33.405 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-02T12:28:33.446 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-02T12:28:33.518 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-02T12:28:33.580 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-02T12:28:33.627 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-02T12:28:33.683 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-02T12:28:33.739 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-02T12:28:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-02T12:28:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-02T12:28:34.015 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-02T12:28:34.063 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-02T12:28:34.093 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-02T12:28:34.170 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-02T12:28:34.212 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-02T12:28:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-02T12:28:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-02T12:28:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-02T12:28:34.372 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-02T12:28:34.462 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-02T12:28:34.512 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-02T12:28:34.583 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-02T12:28:34.720 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-02T12:28:34.846 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-02T12:28:35.051 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2589.g38caedd8.el9. 31/36 2024-04-02T12:28:35.076 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 32/36 2024-04-02T12:28:35.102 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 32/36 2024-04-02T12:28:35.102 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:28:35.102 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T12:28:35.102 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T12:28:35.102 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T12:28:35.102 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:45.900 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 33/36 2024-04-02T12:28:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 33/36 2024-04-02T12:28:45.928 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:28:45.928 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T12:28:45.928 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T12:28:45.928 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T12:28:45.928 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:46.459 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 34/36 2024-04-02T12:28:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 34/36 2024-04-02T12:28:46.483 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:28:46.483 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T12:28:46.483 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T12:28:46.483 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T12:28:46.483 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:46.718 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 35/36 2024-04-02T12:28:46.741 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 35/36 2024-04-02T12:28:46.741 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:28:46.741 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T12:28:46.741 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T12:28:46.741 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T12:28:46.741 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:46.807 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 36/36 2024-04-02T12:28:53.676 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 36/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 1/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 2/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 3/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 4/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 5/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2589.g38caedd8.el9. 6/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-02T12:28:53.677 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-02T12:28:53.678 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-02T12:28:53.679 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-02T12:28:54.728 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-02T12:28:54.728 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:54.728 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:28:54.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T12:28:54.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T12:28:54.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:54.731 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:28:54.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2024-04-02T12:28:55.531 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:53 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:28:55.631 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:28:55.684 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:28:55.685 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:28:55.685 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:28:55.747 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2024-04-02T12:28:56.314 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:54 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:28:56.465 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 762 k 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:28:56.466 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:28:56.467 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:28:56.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:56.467 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 762 k 2024-04-02T12:28:56.467 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 768 k 2024-04-02T12:28:56.467 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:28:56.818 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-19.0.0-2589.g38caedd8.el9.noarch.rpm 2.1 MB/s | 762 kB 00:00 2024-04-02T12:28:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:28:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.1 MB/s | 762 kB 00:00 2024-04-02T12:28:56.819 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:28:56.824 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:28:56.824 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:28:56.830 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:28:56.830 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:28:56.903 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:28:57.278 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:28:57.373 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:28:57.719 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:28:58.001 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/1 2024-04-02T12:28:58.001 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:58.001 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:28:58.001 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:28:58.001 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:58.001 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:28:58.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2024-04-02T12:28:58.822 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:56 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:28:58.977 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:28:58.978 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:28:58.978 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 157 k 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 157 k 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 640 k 2024-04-02T12:28:58.979 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:28:59.209 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-19.0.0-2589.g38caed 686 kB/s | 157 kB 00:00 2024-04-02T12:28:59.210 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:28:59.210 INFO:teuthology.orchestra.run.smithi129.stdout:Total 682 kB/s | 157 kB 00:00 2024-04-02T12:28:59.210 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:28:59.219 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:28:59.220 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:28:59.272 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:28:59.272 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:28:59.419 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:28:59.477 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8. 1/1 2024-04-02T12:28:59.502 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8. 1/1 2024-04-02T12:28:59.502 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:28:59.502 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T12:28:59.502 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:00.593 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8. 1/1 2024-04-02T12:29:00.593 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:00.593 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:00.593 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:29:00.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:00.594 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:00.842 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2024-04-02T12:29:01.415 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:59 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:01.514 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:01.566 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:01.567 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:01.567 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:01.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2024-04-02T12:29:02.192 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:00 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:02.345 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 3.5 M 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 34 k 2024-04-02T12:29:02.346 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 16 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-04-02T12:29:02.347 INFO:teuthology.orchestra.run.smithi129.stdout:Install 11 Packages 2024-04-02T12:29:02.348 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:02.348 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 7.6 M 2024-04-02T12:29:02.348 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 100 M 2024-04-02T12:29:02.348 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:02.914 INFO:teuthology.orchestra.run.smithi129.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2589.g38c 87 kB/s | 16 kB 00:00 2024-04-02T12:29:03.014 INFO:teuthology.orchestra.run.smithi129.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2589.g38 117 kB/s | 34 kB 00:00 2024-04-02T12:29:03.039 INFO:teuthology.orchestra.run.smithi129.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-04-02T12:29:03.315 INFO:teuthology.orchestra.run.smithi129.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-04-02T12:29:03.365 INFO:teuthology.orchestra.run.smithi129.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-02T12:29:03.432 INFO:teuthology.orchestra.run.smithi129.stdout:(6/11): ceph-mgr-dashboard-19.0.0-2589.g38caedd 5.0 MB/s | 3.5 MB 00:00 2024-04-02T12:29:03.532 INFO:teuthology.orchestra.run.smithi129.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-04-02T12:29:03.583 INFO:teuthology.orchestra.run.smithi129.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 613 kB/s | 31 kB 00:00 2024-04-02T12:29:03.641 INFO:teuthology.orchestra.run.smithi129.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-04-02T12:29:03.666 INFO:teuthology.orchestra.run.smithi129.stdout:(10/11): python3-grpcio-tools-1.46.7-10.el9.x86 617 kB/s | 144 kB 00:00 2024-04-02T12:29:04.725 INFO:teuthology.orchestra.run.smithi129.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 505 kB/s | 863 kB 00:01 2024-04-02T12:29:04.729 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:04.730 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.2 MB/s | 7.6 MB 00:02 2024-04-02T12:29:04.750 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:04.768 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:04.769 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:04.940 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:04.940 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:05.606 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-02T12:29:05.645 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-02T12:29:05.762 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-02T12:29:05.896 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-02T12:29:05.948 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-02T12:29:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-02T12:29:06.086 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-02T12:29:06.113 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-02T12:29:06.145 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2589.g38caedd8.el9 9/11 2024-04-02T12:29:07.183 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2589.g38caedd8.el 10/11 2024-04-02T12:29:07.221 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noa 11/11 2024-04-02T12:29:08.547 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noa 11/11 2024-04-02T12:29:08.547 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2589.g38caedd8.el 1/11 2024-04-02T12:29:08.547 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noa 2/11 2024-04-02T12:29:08.547 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2589.g38caedd8.el9 3/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-02T12:29:08.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:08.896 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:08.897 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:09.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-02T12:29:09.754 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:07 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:09.906 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:09.907 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 7.4 M 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T12:29:09.908 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-04-02T12:29:09.909 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:09.910 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 37 M 2024-04-02T12:29:09.910 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 209 M 2024-04-02T12:29:09.910 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:10.251 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 193 kB/s | 32 kB 00:00 2024-04-02T12:29:10.293 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-04-02T12:29:10.335 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 817 kB/s | 34 kB 00:00 2024-04-02T12:29:10.510 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.1 MB/s | 3.0 MB 00:00 2024-04-02T12:29:10.769 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-04-02T12:29:10.928 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-25 8.8 MB/s | 7.4 MB 00:00 2024-04-02T12:29:11.006 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.7 MB/s | 4.8 MB 00:00 2024-04-02T12:29:11.037 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 818 kB/s | 217 kB 00:00 2024-04-02T12:29:11.321 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 691 kB/s | 195 kB 00:00 2024-04-02T12:29:11.588 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-02T12:29:11.613 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 846 kB/s | 245 kB 00:00 2024-04-02T12:29:11.730 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-02T12:29:11.732 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:11.732 INFO:teuthology.orchestra.run.smithi129.stdout:Total 20 MB/s | 37 MB 00:01 2024-04-02T12:29:11.773 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:11.798 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:11.798 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:12.153 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:12.153 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:12.870 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:12.944 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-02T12:29:12.975 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-02T12:29:13.069 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-02T12:29:13.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-02T12:29:13.306 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-02T12:29:13.344 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-02T12:29:13.430 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-02T12:29:14.073 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-02T12:29:14.182 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-02T12:29:15.406 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-02T12:29:16.437 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-02T12:29:16.462 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2589.g38cae 12/12 2024-04-02T12:29:18.937 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2589.g38cae 12/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2589.g38cae 1/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-02T12:29:18.938 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-02T12:29:19.366 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T12:29:19.367 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T12:29:19.368 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T12:29:19.368 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T12:29:19.368 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:19.368 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:19.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2024-04-02T12:29:20.391 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:18 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:20.542 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 49 k 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T12:29:20.543 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2024-04-02T12:29:20.544 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:20.545 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.0 M 2024-04-02T12:29:20.545 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-04-02T12:29:20.545 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:20.740 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 209 kB/s | 24 kB 00:00 2024-04-02T12:29:20.757 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-04-02T12:29:20.815 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-02T12:29:20.841 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-04-02T12:29:20.866 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-02T12:29:20.883 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-04-02T12:29:20.900 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): ceph-mgr-rook-19.0.0-2589.g38caedd8.el9 178 kB/s | 49 kB 00:00 2024-04-02T12:29:21.134 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-04-02T12:29:21.150 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-04-02T12:29:21.284 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-04-02T12:29:21.318 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 485 kB/s | 203 kB 00:00 2024-04-02T12:29:21.393 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 825 kB/s | 90 kB 00:00 2024-04-02T12:29:21.501 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-04-02T12:29:21.506 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:21.506 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-04-02T12:29:21.506 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:21.519 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:21.519 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:21.609 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:21.611 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:22.119 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:22.288 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-02T12:29:22.353 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-02T12:29:22.424 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-02T12:29:22.462 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-02T12:29:22.503 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-02T12:29:22.606 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-02T12:29:22.715 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-02T12:29:22.772 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-02T12:29:23.497 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-02T12:29:23.566 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-02T12:29:23.595 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-02T12:29:23.652 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-02T12:29:23.682 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 13/13 2024-04-02T12:29:24.536 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 13/13 2024-04-02T12:29:24.536 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 1/13 2024-04-02T12:29:24.536 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-02T12:29:24.537 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T12:29:24.912 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:24.913 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:25.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2024-04-02T12:29:25.724 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:23 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:25.879 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:25.879 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-04-02T12:29:25.879 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:29:25.879 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 141 k 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-04-02T12:29:25.880 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:25.881 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 6.8 M 2024-04-02T12:29:25.881 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2024-04-02T12:29:25.881 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:26.331 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-04-02T12:29:26.397 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2589.g38caedd8.e 518 kB/s | 141 kB 00:00 2024-04-02T12:29:26.481 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-04-02T12:29:26.899 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 116 kB/s | 58 kB 00:00 2024-04-02T12:29:27.031 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 101 kB/s | 51 kB 00:00 2024-04-02T12:29:27.173 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 651 kB/s | 548 kB 00:00 2024-04-02T12:29:27.177 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:27.177 INFO:teuthology.orchestra.run.smithi129.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2024-04-02T12:29:27.190 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:27.201 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:27.201 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:27.292 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:27.293 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:27.589 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:27.755 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-02T12:29:27.808 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-02T12:29:28.331 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-02T12:29:28.411 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-02T12:29:28.482 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-02T12:29:28.521 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 6/6 2024-04-02T12:29:29.336 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 6/6 2024-04-02T12:29:29.336 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 1/6 2024-04-02T12:29:29.336 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-02T12:29:29.337 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-02T12:29:29.337 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-02T12:29:29.337 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T12:29:29.658 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T12:29:29.659 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T12:29:29.659 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:29.659 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:29.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2024-04-02T12:29:30.495 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:28 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 871 k 2024-04-02T12:29:30.651 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:30.652 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 950 k 2024-04-02T12:29:30.653 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 4.0 M 2024-04-02T12:29:30.653 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:30.785 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 599 kB/s | 79 kB 00:00 2024-04-02T12:29:31.109 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-fuse-19.0.0-2589.g38caedd8.el9.x86_ 1.9 MB/s | 871 kB 00:00 2024-04-02T12:29:31.111 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:31.111 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.0 MB/s | 950 kB 00:00 2024-04-02T12:29:31.111 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:31.134 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:31.134 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:31.207 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:31.207 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:31.391 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:31.553 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-02T12:29:31.627 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 2/2 2024-04-02T12:29:32.315 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 2/2 2024-04-02T12:29:32.315 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/2 2024-04-02T12:29:32.612 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-02T12:29:32.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:32.612 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:32.612 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-02T12:29:32.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:32.613 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:32.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-volume 2024-04-02T12:29:33.453 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:31 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:33.610 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-2589.g38caedd8.el9 ceph-noarch 264 k 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:29:33.611 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 332 k 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.5 M 2024-04-02T12:29:33.612 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:33.734 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 567 kB/s | 69 kB 00:00 2024-04-02T12:29:34.034 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-volume-19.0.0-2589.g38caedd8.el9.no 625 kB/s | 264 kB 00:00 2024-04-02T12:29:34.035 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:34.036 INFO:teuthology.orchestra.run.smithi129.stdout:Total 784 kB/s | 332 kB 00:00 2024-04-02T12:29:34.036 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:34.044 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:34.044 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:34.082 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:34.083 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:34.492 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-02T12:29:34.529 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 2/2 2024-04-02T12:29:34.546 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 2/2 2024-04-02T12:29:34.546 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:29:34.546 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T12:29:34.546 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:35.264 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 1/2 2024-04-02T12:29:35.591 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-02T12:29:35.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:35.592 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:35.592 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-2589.g38caedd8.el9.noarch 2024-04-02T12:29:35.592 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T12:29:35.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:35.592 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:35.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2024-04-02T12:29:36.476 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:34 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:36.627 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:36.628 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:36.628 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-04-02T12:29:36.628 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:36.628 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:36.628 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 125 k 2024-04-02T12:29:36.628 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 125 k 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 445 k 2024-04-02T12:29:36.629 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:36.850 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-19.0.0-2589.g38caedd8.el9.x86_64 565 kB/s | 125 kB 00:00 2024-04-02T12:29:36.851 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:36.851 INFO:teuthology.orchestra.run.smithi129.stdout:Total 561 kB/s | 125 kB 00:00 2024-04-02T12:29:36.851 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:36.856 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:36.857 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:36.914 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:36.914 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:37.058 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:37.157 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:37.616 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:37.916 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:37.917 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:29:37.917 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:37.917 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:38.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2024-04-02T12:29:38.825 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:36 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:38.929 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:38.984 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:38.985 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:38.985 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:39.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2024-04-02T12:29:39.617 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:37 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:39.767 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 32 k 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:39.768 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 32 k 2024-04-02T12:29:39.769 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 142 k 2024-04-02T12:29:39.769 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:39.947 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-19.0.0-2589.g38caedd8.el9.x86_6 177 kB/s | 32 kB 00:00 2024-04-02T12:29:39.948 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:39.948 INFO:teuthology.orchestra.run.smithi129.stdout:Total 176 kB/s | 32 kB 00:00 2024-04-02T12:29:39.948 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:39.951 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:39.952 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:39.978 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:39.978 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:40.071 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:40.170 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:40.574 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:40.841 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:40.841 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:40.841 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:40.841 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:29:40.841 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:40.841 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:41.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2024-04-02T12:29:41.568 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:39 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:41.671 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:41.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2024-04-02T12:29:42.356 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:40 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:42.455 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:42.508 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:42.509 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:42.509 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:42.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2024-04-02T12:29:43.141 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:41 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:43.240 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:43.292 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:43.293 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:43.293 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:43.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2024-04-02T12:29:43.925 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:41 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:44.023 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:44.075 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:44.076 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:44.076 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:44.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2024-04-02T12:29:44.697 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:42 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:44.795 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:44.846 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:44.847 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:44.847 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:44.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2024-04-02T12:29:45.482 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:43 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:45.582 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:19.0.0-2589.g38caedd8.el9.x86_64 is already installed. 2024-04-02T12:29:45.634 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:45.635 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-04-02T12:29:45.635 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:45.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2024-04-02T12:29:46.256 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:44 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:46.408 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 90 k 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 90 k 2024-04-02T12:29:46.409 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 313 k 2024-04-02T12:29:46.410 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:46.626 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-19.0.0-2589.g38caedd8.el9.x86_64.rpm 415 kB/s | 90 kB 00:00 2024-04-02T12:29:46.627 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:46.627 INFO:teuthology.orchestra.run.smithi129.stdout:Total 413 kB/s | 90 kB 00:00 2024-04-02T12:29:46.627 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:46.635 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:46.636 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:46.690 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:46.691 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:46.833 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:46.941 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:47.370 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:47.640 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:47.640 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:47.640 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:47.640 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:29:47.640 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:47.640 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:47.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2024-04-02T12:29:48.463 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:46 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:48.623 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:48.624 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:48.624 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-04-02T12:29:48.624 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:48.624 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 3.4 M 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.4 M 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 29 M 2024-04-02T12:29:48.625 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:49.014 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-19.0.0-2589.g38caedd8.el9.x86_64.rpm 8.8 MB/s | 3.4 MB 00:00 2024-04-02T12:29:49.014 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:49.015 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.7 MB/s | 3.4 MB 00:00 2024-04-02T12:29:49.015 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:49.026 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:49.026 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:49.101 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:49.101 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:49.685 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:49.740 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:49.765 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:49.765 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T12:29:49.766 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T12:29:49.766 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T12:29:49.766 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T12:29:49.766 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:51.134 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:51.134 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:51.134 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:51.134 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:29:51.134 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:51.134 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:51.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2024-04-02T12:29:52.015 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:50 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:29:52.168 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:29:52.169 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:52.169 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-04-02T12:29:52.169 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:52.169 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:29:52.169 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 176 k 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 176 k 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 673 k 2024-04-02T12:29:52.170 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:29:52.402 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-19.0.0-2589.g38caedd8.el9.x86_64.rpm 758 kB/s | 176 kB 00:00 2024-04-02T12:29:52.403 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:29:52.403 INFO:teuthology.orchestra.run.smithi129.stdout:Total 753 kB/s | 176 kB 00:00 2024-04-02T12:29:52.403 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:29:52.413 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:29:52.413 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:29:52.469 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:29:52.469 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:29:52.618 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:29:52.726 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:53.209 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:53.504 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 1/1 2024-04-02T12:29:53.505 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:53.505 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:29:53.505 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:29:53.505 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:53.505 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:29:53.769 DEBUG:teuthology.parallel:result is None 2024-04-02T12:29:53.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=38caedd896c6b19ed5e43655372d95dc8e78cfda 2024-04-02T12:29:54.023 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T12:29:54.044 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-2589.g38caedd8.el9 2024-04-02T12:29:54.044 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2589.g38caedd8.el9 2024-04-02T12:29:54.044 INFO:teuthology.task.install:The correct ceph version 19.0.0-2589.g38caedd8 is installed. 2024-04-02T12:29:54.046 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T12:29:54.046 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:54.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T12:29:54.112 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T12:29:54.112 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:54.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T12:29:54.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T12:29:54.246 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T12:29:54.246 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:54.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T12:29:54.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T12:29:54.379 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T12:29:54.380 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:54.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T12:29:54.453 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T12:29:54.521 INFO:teuthology.run_tasks:Running task ceph... 2024-04-02T12:29:54.617 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-02T12:29:54.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-04-02T12:29:54.644 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-02T12:29:54.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T12:29:54.711 INFO:tasks.ceph:Creating extra log directories... 2024-04-02T12:29:54.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T12:29:54.804 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-02T12:29:54.804 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-02T12:29:54.805 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527', 'branch': 'wip-matanb-crimson-testing-apr-2', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7635527', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-apr-2'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '107e1624f2a9b72a56ee529969558944d24a2a75'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 223, 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_107e1624f2a9b72a56ee529969558944d24a2a75/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '107e1624f2a9b72a56ee529969558944d24a2a75', 'targets': {'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN7cKe3M/hlEtYGKZhQBrbfmzpghFhhXH2QRAGfr8SGEZFK+46HggvnvpGb+1Xq0cwXjhWfCwMsCX7CukabtRAI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38caedd896c6b19ed5e43655372d95dc8e78cfda', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_11:03:04', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-02T12:29:54.805 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T12:29:54.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T12:29:54.900 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:54.901 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T12:29:54.956 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T12:29:54.956 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-04-02T12:29:55.011 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T12:29:55.011 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-02 12:29:50.040949871 +0000 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-02 12:24:07.390659786 +0000 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-02 12:24:07.390659786 +0000 2024-04-02T12:29:55.012 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-02 12:24:07.390659786 +0000 2024-04-02T12:29:55.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T12:29:55.077 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-02T12:29:55.077 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-02T12:29:55.077 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000205606 s, 2.5 MB/s 2024-04-02T12:29:55.078 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T12:29:55.134 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-04-02T12:29:55.189 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T12:29:55.189 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:29:55.189 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-02T12:29:55.190 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:29:55.190 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:29:55.190 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-02 12:29:50.040949871 +0000 2024-04-02T12:29:55.190 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-02 12:24:07.760651595 +0000 2024-04-02T12:29:55.190 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-02 12:24:07.760651595 +0000 2024-04-02T12:29:55.190 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-02 12:24:07.760651595 +0000 2024-04-02T12:29:55.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T12:29:55.255 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-02T12:29:55.255 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-02T12:29:55.255 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000185371 s, 2.8 MB/s 2024-04-02T12:29:55.256 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T12:29:55.312 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-04-02T12:29:55.367 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T12:29:55.367 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:29:55.367 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-02T12:29:55.367 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:29:55.367 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:29:55.367 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-02 12:29:50.040949871 +0000 2024-04-02T12:29:55.368 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-02 12:24:08.102644023 +0000 2024-04-02T12:29:55.368 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-02 12:24:08.102644023 +0000 2024-04-02T12:29:55.368 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-02 12:24:08.102644023 +0000 2024-04-02T12:29:55.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T12:29:55.433 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-02T12:29:55.433 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-02T12:29:55.433 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000212681 s, 2.4 MB/s 2024-04-02T12:29:55.434 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T12:29:55.491 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-04-02T12:29:55.545 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T12:29:55.545 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T12:29:55.545 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-02T12:29:55.545 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T12:29:55.545 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T12:29:55.545 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-02 12:29:50.041949849 +0000 2024-04-02T12:29:55.546 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-02 12:24:08.443636474 +0000 2024-04-02T12:29:55.546 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-02 12:24:08.443636474 +0000 2024-04-02T12:29:55.546 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-02 12:24:08.443636474 +0000 2024-04-02T12:29:55.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T12:29:55.611 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-02T12:29:55.611 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-02T12:29:55.611 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000199069 s, 2.6 MB/s 2024-04-02T12:29:55.612 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T12:29:55.668 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:29:55.668 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-02T12:29:55.668 INFO:tasks.ceph:Generating config... 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-02T12:29:55.669 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] log to file = False 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-02T12:29:55.670 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-02T12:29:55.671 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-02T12:29:55.671 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-02T12:29:55.671 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-02T12:29:55.671 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] auth client required = none 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] auth service required = none 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] auth supported = none 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-02T12:29:55.671 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-02T12:29:55.672 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-02T12:29:55.673 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-02T12:29:55.674 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-02T12:29:55.675 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-02T12:29:55.675 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-02T12:29:55.676 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-02T12:29:55.676 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-02T12:29:55.676 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-02T12:29:55.676 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-02T12:29:55.676 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-02T12:29:55.676 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-02T12:29:55.676 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-02T12:29:55.676 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-02T12:29:55.676 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-02T12:29:55.676 INFO:tasks.ceph:Setting up mon.a... 2024-04-02T12:29:55.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-02T12:29:55.769 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2024-04-02T12:29:55.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-02T12:29:55.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T12:29:55.852 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129')] 2024-04-02T12:29:55.852 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-02T12:29:55.853 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:55.854 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T12:29:55.907 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:29:55.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:29:56.006+0000 7f23f0874f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:29:55.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:29:56.006+0000 7f23f0874f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:29:55.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:29:56.006+0000 7f23f0874f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:29:55.999 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:29:55.999 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid 6d7dadcc-da45-4253-800e-c932fcf12369 2024-04-02T12:29:55.999 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:fsid 6d7dadcc-da45-4253-800e-c932fcf12369 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2024-04-02T12:29:56.007995+0000 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-04-02T12:29:56.007995+0000 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 17 (quincy) 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 1 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-04-02T12:29:56.000 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-02T12:29:56.001 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T12:29:56.015 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6d7dadcc-da45-4253-800e-c932fcf12369... 2024-04-02T12:29:56.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T12:29:56.105 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = none 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-04-02T12:29:56.106 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-04-02T12:29:56.107 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-04-02T12:29:56.108 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = 172.21.15.129 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: auth client required = none 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: auth cluster required = none 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: auth service required = none 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug asok = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug bdev = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug buffer = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug civetweb = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug client = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug compressor = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug context = 0/0 2024-04-02T12:29:56.109 INFO:teuthology.orchestra.run.smithi129.stdout: debug crush = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug crypto = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug dpdk = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug eventtrace = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug filer = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug filestore = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug finisher = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug fuse = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug heartbeatmap = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug javaclient = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug journal = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug journaler = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug kstore = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug leveldb = 0/0 2024-04-02T12:29:56.110 INFO:teuthology.orchestra.run.smithi129.stdout: debug lockdep = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds balancer = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds locker = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds log = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds log expire = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mds migrator = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug memdb = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgrc = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug objclass = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug objectcacher = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug objecter = 0/0 2024-04-02T12:29:56.111 INFO:teuthology.orchestra.run.smithi129.stdout: debug optracker = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug perfcounter = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rados = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd mirror = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rbd replay = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug refs = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rgw = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rgw lifecycle = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rgw notification = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rgw sync = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 0/0 2024-04-02T12:29:56.112 INFO:teuthology.orchestra.run.smithi129.stdout: debug striper = 0/0 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: debug throttle = 0/0 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: debug timer = 0/0 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: debug tp = 0/0 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: ms client mode = crc 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: ms cluster mode = crc 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: ms mon client mode = crc 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: ms mon cluster mode = crc 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: ms mon service mode = crc 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: ms service mode = crc 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 6d7dadcc-da45-4253-800e-c932fcf12369 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.113 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-04-02T12:29:56.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-04-02T12:29:56.115 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug monc = 20 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression algorithm = snappy 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore compression mode = aggressive 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore rocksdb cf = False 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: crimson alien thread cpu cores = 6-7 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: crimson osd obc lru size = 10 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 20 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 20 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug filestore = 0/0 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug journal = 0/0 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 0/0 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 0/0 2024-04-02T12:29:56.116 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 10 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: err to stderr = True 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: log flush on exit = True 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: log to file = False 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: log to stderr = True 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: osd shutdown pgref assert = True 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-04-02T12:29:56.117 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 0/0 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 0/0 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 0/0 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-04-02T12:29:56.118 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.119 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default crimson = True 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 0-2 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 3-5 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 0-2 2024-04-02T12:29:56.120 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2024-04-02T12:29:56.121 INFO:teuthology.orchestra.run.smithi129.stdout: crimson seastar cpu cores = 3-5 2024-04-02T12:29:56.122 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-02T12:29:56.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-02T12:29:56.188 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-02T12:29:56.188 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:56.189 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-02T12:29:56.204 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:56.204 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-02T12:29:56.258 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:29:56.258 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:56.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T12:29:56.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T12:29:56.337 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:29:56.337 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:29:56.392 INFO:tasks.ceph:Setting up mon nodes... 2024-04-02T12:29:56.392 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-02T12:29:56.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T12:29:56.496 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T12:29:56.499 INFO:tasks.ceph:Setting up mds nodes... 2024-04-02T12:29:56.499 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-02T12:29:56.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T12:29:56.550 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-02T12:29:56.565 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-02T12:29:56.565 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-02T12:29:56.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-02T12:29:56.593 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:29:56.593 INFO:tasks.ceph:role: osd.0 2024-04-02T12:29:56.593 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:29:56.593 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T12:29:56.666 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T12:29:56.666 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T12:29:56.667 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T12:29:57.168 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-04-02T12:29:57.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-04-02T12:29:57.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-02T12:29:57.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-02T12:29:57.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-02T12:29:57.415 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:29:57.415 INFO:tasks.ceph:role: osd.1 2024-04-02T12:29:57.416 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:29:57.416 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T12:29:57.512 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T12:29:57.513 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T12:29:57.513 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T12:29:57.956 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-04-02T12:29:57.958 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-04-02T12:29:57.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-02T12:29:58.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-02T12:29:58.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-02T12:29:58.148 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T12:29:58.149 INFO:tasks.ceph:role: osd.2 2024-04-02T12:29:58.149 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:29:58.149 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T12:29:58.223 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T12:29:58.223 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T12:29:58.223 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T12:29:58.223 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T12:29:58.223 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T12:29:58.223 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-04-02T12:29:58.224 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T12:29:58.224 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T12:29:58.224 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T12:29:58.224 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T12:29:58.709 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-04-02T12:29:58.711 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-04-02T12:29:58.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-02T12:29:58.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-02T12:29:58.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:29:59.204 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:29:59.205 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-04-02 12:29:59,216 seastar - Reactor backend: linux-aio 2024-04-02T12:29:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:==28802==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:29:59.356 INFO:teuthology.orchestra.run.smithi129.stderr:DEBUG 2024-04-02 12:29:59,366 [shard 0:main] seastar - reactor::drain 2024-04-02T12:29:59.899 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:29:59.899 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-04-02T12:29:59.899 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-04-02T12:29:59.899 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-04-02T12:29:59.899 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:29:59.899 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:29:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:29:59.960 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-04-02 12:29:59,971 seastar - Reactor backend: linux-aio 2024-04-02T12:29:59.969 INFO:teuthology.orchestra.run.smithi129.stderr:==28800==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:04.614 INFO:teuthology.orchestra.run.smithi129.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 6d7dadcc-da45-4253-800e-c932fcf12369 2024-04-02T12:30:04.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T12:30:04.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:30:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:30:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-04-02 12:30:04,963 seastar - Reactor backend: linux-aio 2024-04-02T12:30:04.957 INFO:teuthology.orchestra.run.smithi129.stderr:==28985==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:30:05.090 INFO:teuthology.orchestra.run.smithi129.stderr:DEBUG 2024-04-02 12:30:05,100 [shard 0:main] seastar - reactor::drain 2024-04-02T12:30:05.645 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:05.645 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-04-02T12:30:05.645 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-04-02T12:30:05.645 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-04-02T12:30:05.645 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:05.645 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:05.701 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:30:05.705 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-04-02 12:30:05,715 seastar - Reactor backend: linux-aio 2024-04-02T12:30:05.724 INFO:teuthology.orchestra.run.smithi129.stderr:==28983==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:30:10.350 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:10.351 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-04-02T12:30:10.351 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-04-02T12:30:10.351 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-04-02T12:30:10.351 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:10.351 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:10.351 INFO:teuthology.orchestra.run.smithi129.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 6d7dadcc-da45-4253-800e-c932fcf12369 2024-04-02T12:30:10.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T12:30:10.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:30:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:30:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-04-02 12:30:10,700 seastar - Reactor backend: linux-aio 2024-04-02T12:30:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:==29167==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:30:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:DEBUG 2024-04-02 12:30:10,837 [shard 0:main] seastar - reactor::drain 2024-04-02T12:30:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-04-02T12:30:11.358 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-04-02T12:30:11.358 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-04-02T12:30:11.358 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:11.358 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:11.416 INFO:teuthology.orchestra.run.smithi129.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T12:30:11.422 INFO:teuthology.orchestra.run.smithi129.stderr:INFO 2024-04-02 12:30:11,430 seastar - Reactor backend: linux-aio 2024-04-02T12:30:11.433 INFO:teuthology.orchestra.run.smithi129.stderr:==29165==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T12:30:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9de8c9 0xb8b1390 0xb8b290c 0xb8b2b22 0xb8b2c78 0xb8b3141 0x54daf 0x174af 0x2eab 0x4a09d 0x35b7c 0x3629e 0x59fd1b8 0x5a0020d 0x5c5af5f 0x5c5b214 0x5c057c1 0x5c08467 0x5c0907e 0x6ee1bf4 0x6ef33c7 0x6ef4864 0x6ef48e9 0x6ef49eb 0x6ef4ae3 0x6ef4cf0 0x6ef4ed6 0x6ef5225 0x6ef55c9 0x6ef57f9 0x6ef5a63 0x6ef5d30 0x6ef5f4e 0x6ef6006 0x6ef60cc 0x6ef61ad 0x5a54aa8 0x5a54bb2 0x5a54c6e 0x5a54d2a 0x5a54de6 0x5a54f33 0x5a550be 0x5a5547f 0x5a71efc 0xb8c4015 0xb8de2af 0xb97f2ad 0xb9808fb 0xb65d063 0xb65d9df 0x368b58a 0x3feaf 0x3ff5f 0x3477444 2024-04-02T12:30:11.498 INFO:teuthology.orchestra.run.smithi129.stderr:kernel callstack: 2024-04-02T12:30:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:Suppressions used: 2024-04-02T12:30:16.068 INFO:teuthology.orchestra.run.smithi129.stderr: count bytes template 2024-04-02T12:30:16.069 INFO:teuthology.orchestra.run.smithi129.stderr: 1 8 MallocExtension::Register 2024-04-02T12:30:16.069 INFO:teuthology.orchestra.run.smithi129.stderr:----------------------------------------------------- 2024-04-02T12:30:16.069 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:16.069 INFO:teuthology.orchestra.run.smithi129.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 6d7dadcc-da45-4253-800e-c932fcf12369 2024-04-02T12:30:16.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T12:30:16.118 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-02T12:30:16.118 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:16.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-02T12:30:16.185 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:16.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-02T12:30:16.252 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:16.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-02T12:30:16.319 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:16.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-02T12:30:16.386 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:16.386 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-02T12:30:16.441 INFO:tasks.ceph:Adding keys to all mons... 2024-04-02T12:30:16.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T12:30:16.506 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-04-02T12:30:16.506 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBE+gtmsgI2HhAA63/nRAtuuUuHYH97Me5J4g== 2024-04-02T12:30:16.506 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-04-02T12:30:16.506 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBI+gtmscmGBxAAOeT1B1IfLW0ROjQ/s9F/vw== 2024-04-02T12:30:16.506 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-04-02T12:30:16.506 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBN+gtmpiC+NBAAnFKCkPchJP9+pcr0f/3ZcQ== 2024-04-02T12:30:16.507 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-04-02T12:30:16.507 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBT+gtmzjNEIxAAYHKJXSbjmEUVzkbWfaep7g== 2024-04-02T12:30:16.507 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-04-02T12:30:16.507 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBE+gtmSThrIRAAwiQkXKwJKsYPZEPMQTtfmA== 2024-04-02T12:30:16.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T12:30:16.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T12:30:16.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T12:30:16.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T12:30:16.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T12:30:16.858 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-02T12:30:16.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-02T12:30:16.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T12:30:16.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:17.006+0000 7fc8386aca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:16.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:17.006+0000 7fc8386aca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-02T12:30:17.036 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T12:30:17.091 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-02T12:30:17.091 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-02T12:30:17.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-02T12:30:17.132 INFO:tasks.ceph.mon.a:Started 2024-04-02T12:30:17.132 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-02T12:30:17.133 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-02T12:30:17.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-02T12:30:17.135 INFO:tasks.ceph.mgr.x:Started 2024-04-02T12:30:17.135 INFO:tasks.ceph:Setting crush tunables to default 2024-04-02T12:30:17.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2024-04-02T12:30:17.219 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:17.228+0000 7f3463710000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.219 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-04-02T12:30:17.228+0000 7f941a06fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-04-02T12:30:17.228+0000 7f941a06fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.220 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-04-02T12:30:17.229+0000 7f941a06fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:17.252+0000 7f53f215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:17.259+0000 7f53f215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.318 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-04-02T12:30:17.329 INFO:tasks.ceph:check_enable_crimson: True 2024-04-02T12:30:17.329 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-02T12:30:17.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-02T12:30:17.422 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:17.431+0000 7f3463710000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.422 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:17.432+0000 7f3463710000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:17.443+0000 7faac61ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:17.449+0000 7faac61ba640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:17.536 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-04-02T12:30:17.546+0000 7f9410df4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-02T12:30:17.537 INFO:tasks.ceph.mon.a.smithi129.stderr:Please be aware that this feature is experimental, untested, 2024-04-02T12:30:17.537 INFO:tasks.ceph.mon.a.smithi129.stderr:unsupported, and may result in data corruption, data loss, 2024-04-02T12:30:17.537 INFO:tasks.ceph.mon.a.smithi129.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-02T12:30:17.537 INFO:tasks.ceph.mon.a.smithi129.stderr:feature with important data. 2024-04-02T12:30:17.537 INFO:tasks.ceph.mon.a.smithi129.stderr: 2024-04-02T12:30:17.911 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:17.921+0000 7f3463710000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T12:30:17.963 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:17.973+0000 7f3463710000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T12:30:18.330 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-02T12:30:18.331 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:18.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-02T12:30:18.358 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:18.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-02T12:30:18.426 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:18.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-02T12:30:18.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new dc3870f0-d37b-40a3-8884-a793f553b65a 0 2024-04-02T12:30:18.496 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.505+0000 7f3463710000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T12:30:18.552 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.562+0000 7f3463710000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T12:30:18.599 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.608+0000 7f3463710000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T12:30:18.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:18.608+0000 7f833282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:18.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:18.616+0000 7f833282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:18.653 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.663+0000 7f3463710000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T12:30:18.683 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:18.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 6aa12f92-6799-49a1-909f-391bbff2dcf6 1 2024-04-02T12:30:18.697 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.707+0000 7f3463710000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T12:30:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:18.808+0000 7f6f70024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:18.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:18.815+0000 7f6f70024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:18.819 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.829+0000 7f3463710000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T12:30:18.881 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-04-02T12:30:18.889 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.899+0000 7f3463710000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T12:30:18.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd new 654c5ce2-2690-4b9d-8580-db0555b426b0 2 2024-04-02T12:30:18.936 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:18.945+0000 7f3463710000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T12:30:18.991 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.000+0000 7f3463710000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T12:30:19.040 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.049+0000 7f3463710000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T12:30:19.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:19.046+0000 7f83c862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:19.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:19.054+0000 7f83c862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:19.121 INFO:teuthology.orchestra.run.smithi129.stdout:2 2024-04-02T12:30:19.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-02T12:30:19.205 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-02T12:30:19.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-02T12:30:19.247 INFO:tasks.ceph.osd.0:Started 2024-04-02T12:30:19.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-02T12:30:19.278 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-02T12:30:19.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-02T12:30:19.321 INFO:tasks.ceph.osd.1:Started 2024-04-02T12:30:19.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-02T12:30:19.353 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-02T12:30:19.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-02T12:30:19.396 INFO:tasks.ceph.osd.2:Started 2024-04-02T12:30:19.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T12:30:19.456 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.465+0000 7f3463710000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T12:30:19.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:19.574+0000 7fac3326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:19.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:19.582+0000 7fac3326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:19.627 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.637+0000 7f3463710000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T12:30:19.643 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:19.643 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":6,"fsid":"6d7dadcc-da45-4253-800e-c932fcf12369","created":"2024-04-02T12:30:17.253075+0000","modified":"2024-04-02T12:30:19.130998+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T12:30:19.130998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dc3870f0-d37b-40a3-8884-a793f553b65a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6aa12f92-6799-49a1-909f-391bbff2dcf6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"654c5ce2-2690-4b9d-8580-db0555b426b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:30:19.657 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-02T12:30:19.657 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-02T12:30:19.791 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.800+0000 7f3463710000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T12:30:19.839 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.848+0000 7f3463710000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T12:30:19.957 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:19.967+0000 7f3463710000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T12:30:19.959 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T12:30:20.061 INFO:teuthology.misc.health.smithi129.stderr:2024-04-02T12:30:20.066+0000 7f75f0c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:20.061 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:20.070+0000 7f3463710000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T12:30:20.063 INFO:teuthology.misc.health.smithi129.stderr:2024-04-02T12:30:20.072+0000 7f75f0c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:20.131 INFO:teuthology.misc.health.smithi129.stdout: 2024-04-02T12:30:20.132 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":6,"fsid":"6d7dadcc-da45-4253-800e-c932fcf12369","created":"2024-04-02T12:30:17.253075+0000","modified":"2024-04-02T12:30:19.130998+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T12:30:19.130998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dc3870f0-d37b-40a3-8884-a793f553b65a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6aa12f92-6799-49a1-909f-391bbff2dcf6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"654c5ce2-2690-4b9d-8580-db0555b426b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:30:20.142 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-02T12:30:20.229 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:20.238+0000 7f3463710000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T12:30:20.275 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:20.285+0000 7f3463710000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T12:30:20.445 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:20.454+0000 7f3463710000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T12:30:21.328 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:21.337+0000 7f3463710000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T12:30:21.613 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:21.622+0000 7f3463710000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T12:30:21.665 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:21.675+0000 7f3463710000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T12:30:21.790 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:21.800+0000 7f3463710000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T12:30:22.800 INFO:tasks.ceph.mgr.x.smithi129.stderr:2024-04-02T12:30:22.810+0000 7f3402da9640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-02T12:30:26.445 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T12:30:26.559 INFO:teuthology.misc.health.smithi129.stderr:2024-04-02T12:30:26.563+0000 7fe990f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:26.559 INFO:teuthology.misc.health.smithi129.stderr:2024-04-02T12:30:26.567+0000 7fe990f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:26.745 INFO:teuthology.misc.health.smithi129.stdout: 2024-04-02T12:30:26.745 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":9,"fsid":"6d7dadcc-da45-4253-800e-c932fcf12369","created":"2024-04-02T12:30:17.253075+0000","modified":"2024-04-02T12:30:25.815922+0000","last_up_change":"2024-04-02T12:30:25.815922+0000","last_in_change":"2024-04-02T12:30:19.130998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dc3870f0-d37b-40a3-8884-a793f553b65a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":275872120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":275872120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":275872120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":275872120}]},"public_addr":"172.21.15.129:6805/275872120","cluster_addr":"172.21.15.129:6804/275872120","heartbeat_back_addr":"172.21.15.129:6803/275872120","heartbeat_front_addr":"172.21.15.129:6802/275872120","state":["exists","up"]},{"osd":1,"uuid":"6aa12f92-6799-49a1-909f-391bbff2dcf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":424212014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":424212014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":424212014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":424212014}]},"public_addr":"172.21.15.129:6809/424212014","cluster_addr":"172.21.15.129:6808/424212014","heartbeat_back_addr":"172.21.15.129:6810/424212014","heartbeat_front_addr":"172.21.15.129:6807/424212014","state":["exists","up"]},{"osd":2,"uuid":"654c5ce2-2690-4b9d-8580-db0555b426b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":134153001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":134153001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":134153001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":134153001}]},"public_addr":"172.21.15.129:6811/134153001","cluster_addr":"172.21.15.129:6812/134153001","heartbeat_back_addr":"172.21.15.129:6813/134153001","heartbeat_front_addr":"172.21.15.129:6814/134153001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:30:26.757 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-02T12:30:26.757 INFO:tasks.ceph:Creating RBD pool 2024-04-02T12:30:26.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-02T12:30:26.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:26.881+0000 7fcd44aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:26.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:26.888+0000 7fcd44aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:27.807 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'rbd' created 2024-04-02T12:30:27.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-02T12:30:27.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:27.969+0000 7fc8f5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:27.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:27.981+0000 7fc8f5f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:28.822 INFO:teuthology.orchestra.run.smithi129.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-02T12:30:28.842 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-02T12:30:28.842 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T12:30:28.843 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-02T12:30:28.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-02T12:30:28.843 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-02T12:30:28.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:28.993+0000 7ff6af82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:28.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:28.998+0000 7ff6af82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:29.225 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:29.236 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3133479883},{"type":"v1","addr":"172.21.15.129:6801","nonce":3133479883}]},"active_addr":"172.21.15.129:6801/3133479883","active_change":"2024-04-02T12:30:21.803162+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":52410298}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2447839599}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2429284283}]}]} 2024-04-02T12:30:29.240 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-02T12:30:29.241 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-02T12:30:29.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T12:30:29.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:29.359+0000 7f59b1f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:29.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:29.365+0000 7f59b1f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:29.545 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:29.545 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":12,"fsid":"6d7dadcc-da45-4253-800e-c932fcf12369","created":"2024-04-02T12:30:17.253075+0000","modified":"2024-04-02T12:30:28.830250+0000","last_up_change":"2024-04-02T12:30:25.815922+0000","last_in_change":"2024-04-02T12:30:19.130998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T12:30:25.826088+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-02T12:30:27.077777+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dc3870f0-d37b-40a3-8884-a793f553b65a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":275872120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":275872120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":275872120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":275872120}]},"public_addr":"172.21.15.129:6805/275872120","cluster_addr":"172.21.15.129:6804/275872120","heartbeat_back_addr":"172.21.15.129:6803/275872120","heartbeat_front_addr":"172.21.15.129:6802/275872120","state":["exists","up"]},{"osd":1,"uuid":"6aa12f92-6799-49a1-909f-391bbff2dcf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":424212014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":424212014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":424212014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":424212014}]},"public_addr":"172.21.15.129:6809/424212014","cluster_addr":"172.21.15.129:6808/424212014","heartbeat_back_addr":"172.21.15.129:6810/424212014","heartbeat_front_addr":"172.21.15.129:6807/424212014","state":["exists","up"]},{"osd":2,"uuid":"654c5ce2-2690-4b9d-8580-db0555b426b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":134153001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":134153001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":134153001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":134153001}]},"public_addr":"172.21.15.129:6811/134153001","cluster_addr":"172.21.15.129:6812/134153001","heartbeat_back_addr":"172.21.15.129:6813/134153001","heartbeat_front_addr":"172.21.15.129:6814/134153001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:30:29.558 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-02T12:30:29.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T12:30:29.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:29.676+0000 7f911c2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:29.682+0000 7f911c2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:29.876 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:29.876 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":13,"fsid":"6d7dadcc-da45-4253-800e-c932fcf12369","created":"2024-04-02T12:30:17.253075+0000","modified":"2024-04-02T12:30:29.833486+0000","last_up_change":"2024-04-02T12:30:25.815922+0000","last_in_change":"2024-04-02T12:30:19.130998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T12:30:25.826088+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-02T12:30:27.077777+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dc3870f0-d37b-40a3-8884-a793f553b65a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6805","nonce":275872120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":275872120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6803","nonce":275872120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":275872120}]},"public_addr":"172.21.15.129:6805/275872120","cluster_addr":"172.21.15.129:6804/275872120","heartbeat_back_addr":"172.21.15.129:6803/275872120","heartbeat_front_addr":"172.21.15.129:6802/275872120","state":["exists","up"]},{"osd":1,"uuid":"6aa12f92-6799-49a1-909f-391bbff2dcf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":424212014}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":424212014}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":424212014}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6807","nonce":424212014}]},"public_addr":"172.21.15.129:6809/424212014","cluster_addr":"172.21.15.129:6808/424212014","heartbeat_back_addr":"172.21.15.129:6810/424212014","heartbeat_front_addr":"172.21.15.129:6807/424212014","state":["exists","up"]},{"osd":2,"uuid":"654c5ce2-2690-4b9d-8580-db0555b426b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6811","nonce":134153001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":134153001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":134153001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":134153001}]},"public_addr":"172.21.15.129:6811/134153001","cluster_addr":"172.21.15.129:6812/134153001","heartbeat_back_addr":"172.21.15.129:6813/134153001","heartbeat_front_addr":"172.21.15.129:6814/134153001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T12:30:29.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-02T12:30:29.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-02T12:30:29.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-02T12:30:30.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.013+0000 7f573402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.013+0000 7fea8a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.013+0000 7f498e7f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.020+0000 7f573402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.020+0000 7fea8a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.020+0000 7f498e7f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.035 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-04-02T12:30:30.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:30:30.039 INFO:teuthology.orchestra.run.smithi129.stdout:34359738373 2024-04-02T12:30:30.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:30:30.041 INFO:teuthology.orchestra.run.smithi129.stdout:38654705668 2024-04-02T12:30:30.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T12:30:30.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.155+0000 7f5c0d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.163+0000 7f5c0d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.204+0000 7f6c0719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.205+0000 7fa5b9406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.211+0000 7f6c0719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:30.212+0000 7fa5b9406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:30.361 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:30.374 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.1 2024-04-02T12:30:30.401 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:30.412 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:30.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-02T12:30:30.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-02T12:30:31.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:30:31.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:30:31.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T12:30:31.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:31.507+0000 7f6d97d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:31.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:31.516+0000 7f6d97d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:31.548+0000 7f7bcea17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:31.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:31.555+0000 7f7bcea17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:31.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:31.596+0000 7f7ebaf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:31.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:31.602+0000 7f7ebaf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:31.720 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:31.733 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.1 2024-04-02T12:30:31.742 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:31.755 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-02T12:30:31.789 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:31.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-02T12:30:32.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:30:32.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:30:32.802 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T12:30:32.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:32.869+0000 7fa11be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:32.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:32.878+0000 7fa11be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:32.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:32.909+0000 7f4946d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:32.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:32.918+0000 7f4946d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:32.946+0000 7f7bde9ff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:32.953+0000 7f7bdd79d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:33.084 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:33.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.1 2024-04-02T12:30:33.108 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:33.121 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-02T12:30:33.151 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:33.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-02T12:30:34.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:30:34.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:30:34.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T12:30:34.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:34.222+0000 7f5a06d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:34.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:34.229+0000 7f5a06d1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:34.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:34.269+0000 7f9e1f51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:34.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:34.276+0000 7f9e1f51d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:34.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:34.310+0000 7f5f8d82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:34.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:34.317+0000 7f5f8d82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:34.421 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:34.435 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.1 2024-04-02T12:30:34.466 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:34.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-02T12:30:34.505 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:34.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-02T12:30:35.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:30:35.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:30:35.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T12:30:35.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:35.558+0000 7fa4a211d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:35.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:35.565+0000 7fa4a211d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:35.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:35.603+0000 7fe9a3d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:35.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:35.610+0000 7fe9a3d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:35.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:35.649+0000 7f0c63a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:35.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:35.655+0000 7f0c63a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:35.764 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:35.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.1 2024-04-02T12:30:35.796 INFO:teuthology.orchestra.run.smithi129.stdout:0 2024-04-02T12:30:35.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-02T12:30:35.848 INFO:teuthology.orchestra.run.smithi129.stdout:38654705674 2024-04-02T12:30:35.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705674 for osd.2 2024-04-02T12:30:35.862 DEBUG:teuthology.parallel:result is None 2024-04-02T12:30:36.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T12:30:36.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T12:30:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:36.909+0000 7f6af2206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:36.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:36.920+0000 7f6af2206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:36.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:36.962+0000 7fa51446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:36.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:36.968+0000 7fa51446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:37.115 INFO:teuthology.orchestra.run.smithi129.stdout:38654705673 2024-04-02T12:30:37.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705673 for osd.1 2024-04-02T12:30:37.129 DEBUG:teuthology.parallel:result is None 2024-04-02T12:30:37.161 INFO:teuthology.orchestra.run.smithi129.stdout:34359738378 2024-04-02T12:30:37.174 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738378 for osd.0 2024-04-02T12:30:37.174 DEBUG:teuthology.parallel:result is None 2024-04-02T12:30:37.174 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T12:30:37.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T12:30:37.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:37.332+0000 7f4450d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:37.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:37.337+0000 7f4450d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:37.516 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:37.516 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-02T12:30:37.529 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-02T12:30:35.845406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80936,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034584,"statfs":{"total":289910292480,"available":289827414016,"internally_reserved":0,"allocated":454656,"data_stored":1259777,"data_compressed":59910,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.862171+0000","last_change":"2024-04-02T12:30:28.940276+0000","last_active":"2024-04-02T12:30:29.862171+0000","last_peered":"2024-04-02T12:30:29.862171+0000","last_clean":"2024-04-02T12:30:29.862171+0000","last_became_active":"2024-04-02T12:30:28.938061+0000","last_became_peered":"2024-04-02T12:30:28.938061+0000","last_unstale":"2024-04-02T12:30:29.862171+0000","last_undegraded":"2024-04-02T12:30:29.862171+0000","last_fullsized":"2024-04-02T12:30:29.862171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.863269+0000","last_change":"2024-04-02T12:30:28.943706+0000","last_active":"2024-04-02T12:30:29.863269+0000","last_peered":"2024-04-02T12:30:29.863269+0000","last_clean":"2024-04-02T12:30:29.863269+0000","last_became_active":"2024-04-02T12:30:28.941326+0000","last_became_peered":"2024-04-02T12:30:28.941326+0000","last_unstale":"2024-04-02T12:30:29.863269+0000","last_undegraded":"2024-04-02T12:30:29.863269+0000","last_fullsized":"2024-04-02T12:30:29.863269+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.859417+0000","last_change":"2024-04-02T12:30:28.913574+0000","last_active":"2024-04-02T12:30:29.859417+0000","last_peered":"2024-04-02T12:30:29.859417+0000","last_clean":"2024-04-02T12:30:29.859417+0000","last_became_active":"2024-04-02T12:30:28.909157+0000","last_became_peered":"2024-04-02T12:30:28.909157+0000","last_unstale":"2024-04-02T12:30:29.859417+0000","last_undegraded":"2024-04-02T12:30:29.859417+0000","last_fullsized":"2024-04-02T12:30:29.859417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.862672+0000","last_change":"2024-04-02T12:30:28.947262+0000","last_active":"2024-04-02T12:30:29.862672+0000","last_peered":"2024-04-02T12:30:29.862672+0000","last_clean":"2024-04-02T12:30:29.862672+0000","last_became_active":"2024-04-02T12:30:28.945300+0000","last_became_peered":"2024-04-02T12:30:28.945300+0000","last_unstale":"2024-04-02T12:30:29.862672+0000","last_undegraded":"2024-04-02T12:30:29.862672+0000","last_fullsized":"2024-04-02T12:30:29.862672+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:30.847594+0000","last_change":"2024-04-02T12:30:28.925900+0000","last_active":"2024-04-02T12:30:30.847594+0000","last_peered":"2024-04-02T12:30:30.847594+0000","last_clean":"2024-04-02T12:30:30.847594+0000","last_became_active":"2024-04-02T12:30:28.921789+0000","last_became_peered":"2024-04-02T12:30:28.921789+0000","last_unstale":"2024-04-02T12:30:30.847594+0000","last_undegraded":"2024-04-02T12:30:30.847594+0000","last_fullsized":"2024-04-02T12:30:30.847594+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:30.852831+0000","last_change":"2024-04-02T12:30:28.911944+0000","last_active":"2024-04-02T12:30:30.852831+0000","last_peered":"2024-04-02T12:30:30.852831+0000","last_clean":"2024-04-02T12:30:30.852831+0000","last_became_active":"2024-04-02T12:30:28.909712+0000","last_became_peered":"2024-04-02T12:30:28.909712+0000","last_unstale":"2024-04-02T12:30:30.852831+0000","last_undegraded":"2024-04-02T12:30:30.852831+0000","last_fullsized":"2024-04-02T12:30:30.852831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:30.847417+0000","last_change":"2024-04-02T12:30:28.907368+0000","last_active":"2024-04-02T12:30:30.847417+0000","last_peered":"2024-04-02T12:30:30.847417+0000","last_clean":"2024-04-02T12:30:30.847417+0000","last_became_active":"2024-04-02T12:30:28.900952+0000","last_became_peered":"2024-04-02T12:30:28.900952+0000","last_unstale":"2024-04-02T12:30:30.847417+0000","last_undegraded":"2024-04-02T12:30:30.847417+0000","last_fullsized":"2024-04-02T12:30:30.847417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.861920+0000","last_change":"2024-04-02T12:30:28.892407+0000","last_active":"2024-04-02T12:30:29.861920+0000","last_peered":"2024-04-02T12:30:29.861920+0000","last_clean":"2024-04-02T12:30:29.861920+0000","last_became_active":"2024-04-02T12:30:28.889819+0000","last_became_peered":"2024-04-02T12:30:28.889819+0000","last_unstale":"2024-04-02T12:30:29.861920+0000","last_undegraded":"2024-04-02T12:30:29.861920+0000","last_fullsized":"2024-04-02T12:30:29.861920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.859843+0000","last_change":"2024-04-02T12:30:27.889844+0000","last_active":"2024-04-02T12:30:29.859843+0000","last_peered":"2024-04-02T12:30:29.859843+0000","last_clean":"2024-04-02T12:30:29.859843+0000","last_became_active":"2024-04-02T12:30:27.886393+0000","last_became_peered":"2024-04-02T12:30:27.886393+0000","last_unstale":"2024-04-02T12:30:29.859843+0000","last_undegraded":"2024-04-02T12:30:29.859843+0000","last_fullsized":"2024-04-02T12:30:29.859843+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:26.816790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:26.816790+0000","last_clean_scrub_stamp":"2024-04-02T12:30:26.816790+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":26347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":172032,"data_stored":616715,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616715,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T12:30:37.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T12:30:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:37.646+0000 7fdc1bf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:37.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:37.651+0000 7fdc1bf5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:37.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:37.826 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-04-02T12:30:37.838 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-02T12:30:37.808262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80936,"kb_used_data":444,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034584,"statfs":{"total":289910292480,"available":289827414016,"internally_reserved":0,"allocated":454656,"data_stored":1259777,"data_compressed":59910,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.962855"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.862171+0000","last_change":"2024-04-02T12:30:28.940276+0000","last_active":"2024-04-02T12:30:29.862171+0000","last_peered":"2024-04-02T12:30:29.862171+0000","last_clean":"2024-04-02T12:30:29.862171+0000","last_became_active":"2024-04-02T12:30:28.938061+0000","last_became_peered":"2024-04-02T12:30:28.938061+0000","last_unstale":"2024-04-02T12:30:29.862171+0000","last_undegraded":"2024-04-02T12:30:29.862171+0000","last_fullsized":"2024-04-02T12:30:29.862171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.863269+0000","last_change":"2024-04-02T12:30:28.943706+0000","last_active":"2024-04-02T12:30:29.863269+0000","last_peered":"2024-04-02T12:30:29.863269+0000","last_clean":"2024-04-02T12:30:29.863269+0000","last_became_active":"2024-04-02T12:30:28.941326+0000","last_became_peered":"2024-04-02T12:30:28.941326+0000","last_unstale":"2024-04-02T12:30:29.863269+0000","last_undegraded":"2024-04-02T12:30:29.863269+0000","last_fullsized":"2024-04-02T12:30:29.863269+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.859417+0000","last_change":"2024-04-02T12:30:28.913574+0000","last_active":"2024-04-02T12:30:29.859417+0000","last_peered":"2024-04-02T12:30:29.859417+0000","last_clean":"2024-04-02T12:30:29.859417+0000","last_became_active":"2024-04-02T12:30:28.909157+0000","last_became_peered":"2024-04-02T12:30:28.909157+0000","last_unstale":"2024-04-02T12:30:29.859417+0000","last_undegraded":"2024-04-02T12:30:29.859417+0000","last_fullsized":"2024-04-02T12:30:29.859417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.862672+0000","last_change":"2024-04-02T12:30:28.947262+0000","last_active":"2024-04-02T12:30:29.862672+0000","last_peered":"2024-04-02T12:30:29.862672+0000","last_clean":"2024-04-02T12:30:29.862672+0000","last_became_active":"2024-04-02T12:30:28.945300+0000","last_became_peered":"2024-04-02T12:30:28.945300+0000","last_unstale":"2024-04-02T12:30:29.862672+0000","last_undegraded":"2024-04-02T12:30:29.862672+0000","last_fullsized":"2024-04-02T12:30:29.862672+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:30.847594+0000","last_change":"2024-04-02T12:30:28.925900+0000","last_active":"2024-04-02T12:30:30.847594+0000","last_peered":"2024-04-02T12:30:30.847594+0000","last_clean":"2024-04-02T12:30:30.847594+0000","last_became_active":"2024-04-02T12:30:28.921789+0000","last_became_peered":"2024-04-02T12:30:28.921789+0000","last_unstale":"2024-04-02T12:30:30.847594+0000","last_undegraded":"2024-04-02T12:30:30.847594+0000","last_fullsized":"2024-04-02T12:30:30.847594+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:30.852831+0000","last_change":"2024-04-02T12:30:28.911944+0000","last_active":"2024-04-02T12:30:30.852831+0000","last_peered":"2024-04-02T12:30:30.852831+0000","last_clean":"2024-04-02T12:30:30.852831+0000","last_became_active":"2024-04-02T12:30:28.909712+0000","last_became_peered":"2024-04-02T12:30:28.909712+0000","last_unstale":"2024-04-02T12:30:30.852831+0000","last_undegraded":"2024-04-02T12:30:30.852831+0000","last_fullsized":"2024-04-02T12:30:30.852831+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:30.847417+0000","last_change":"2024-04-02T12:30:28.907368+0000","last_active":"2024-04-02T12:30:30.847417+0000","last_peered":"2024-04-02T12:30:30.847417+0000","last_clean":"2024-04-02T12:30:30.847417+0000","last_became_active":"2024-04-02T12:30:28.900952+0000","last_became_peered":"2024-04-02T12:30:28.900952+0000","last_unstale":"2024-04-02T12:30:30.847417+0000","last_undegraded":"2024-04-02T12:30:30.847417+0000","last_fullsized":"2024-04-02T12:30:30.847417+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.861920+0000","last_change":"2024-04-02T12:30:28.892407+0000","last_active":"2024-04-02T12:30:29.861920+0000","last_peered":"2024-04-02T12:30:29.861920+0000","last_clean":"2024-04-02T12:30:29.861920+0000","last_became_active":"2024-04-02T12:30:28.889819+0000","last_became_peered":"2024-04-02T12:30:28.889819+0000","last_unstale":"2024-04-02T12:30:29.861920+0000","last_undegraded":"2024-04-02T12:30:29.861920+0000","last_fullsized":"2024-04-02T12:30:29.861920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:27.817476+0000","last_clean_scrub_stamp":"2024-04-02T12:30:27.817476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-02T12:30:29.859843+0000","last_change":"2024-04-02T12:30:27.889844+0000","last_active":"2024-04-02T12:30:29.859843+0000","last_peered":"2024-04-02T12:30:29.859843+0000","last_clean":"2024-04-02T12:30:29.859843+0000","last_became_active":"2024-04-02T12:30:27.886393+0000","last_became_peered":"2024-04-02T12:30:27.886393+0000","last_unstale":"2024-04-02T12:30:29.859843+0000","last_undegraded":"2024-04-02T12:30:29.859843+0000","last_fullsized":"2024-04-02T12:30:29.859843+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T12:30:26.816790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T12:30:26.816790+0000","last_clean_scrub_stamp":"2024-04-02T12:30:26.816790+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":26347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":172032,"data_stored":616715,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616715,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29955,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-02T12:30:37.839 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T12:30:37.839 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T12:30:37.839 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-02T12:30:37.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T12:30:37.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:37.957+0000 7f1077a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:37.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:30:37.961+0000 7f1077a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:30:38.180 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:38.180 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T12:30:38.193 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-02T12:30:38.193 INFO:teuthology.run_tasks:Running task cbt... 2024-04-02T12:30:38.207 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi129.front.sepia.ceph.com', 'osds': ['ubuntu@smithi129.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi129.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi129.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi129.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-02T12:30:38.207 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-02T12:30:38.225 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:30:38.225 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-02T12:30:38.278 DEBUG:teuthology.orchestra.run.smithi129:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-02T12:30:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-02T12:30:38.739 INFO:tasks.cbt:Installing collectl 2024-04-02T12:30:38.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-02T12:30:38.841 INFO:teuthology.orchestra.run.smithi129.stderr:--2024-04-02 12:30:38-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-02T12:30:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-04-02T12:30:39.018 INFO:teuthology.orchestra.run.smithi129.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-04-02T12:30:39.238 INFO:teuthology.orchestra.run.smithi129.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-02T12:30:39.238 INFO:teuthology.orchestra.run.smithi129.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmC_pv4UkKk708UNV5qSQk0-0feVtqWc0VQCjX5S0sQ03nEp4rRICrIrdpH_x-6YzXx_I6neiN2sptwRICCeBTxTOXzg%3D%3D&use_mirror=pilotfiber&r= [following] 2024-04-02T12:30:39.238 INFO:teuthology.orchestra.run.smithi129.stderr:--2024-04-02 12:30:39-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmC_pv4UkKk708UNV5qSQk0-0feVtqWc0VQCjX5S0sQ03nEp4rRICrIrdpH_x-6YzXx_I6neiN2sptwRICCeBTxTOXzg%3D%3D&use_mirror=pilotfiber&r= 2024-04-02T12:30:39.252 INFO:teuthology.orchestra.run.smithi129.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-02T12:30:39.320 INFO:teuthology.orchestra.run.smithi129.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-02T12:30:39.585 INFO:teuthology.orchestra.run.smithi129.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-02T12:30:39.585 INFO:teuthology.orchestra.run.smithi129.stderr:Location: https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-02T12:30:39.585 INFO:teuthology.orchestra.run.smithi129.stderr:--2024-04-02 12:30:39-- https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-02T12:30:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:Resolving pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)... 216.158.133.250 2024-04-02T12:30:39.615 INFO:teuthology.orchestra.run.smithi129.stderr:Connecting to pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)|216.158.133.250|:443... connected. 2024-04-02T12:30:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-02T12:30:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-02T12:30:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-02T12:30:39.887 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:39.942 INFO:teuthology.orchestra.run.smithi129.stderr: 0K .......... .......... .......... .......... .......... 8% 914K 1s 2024-04-02T12:30:39.962 INFO:teuthology.orchestra.run.smithi129.stderr: 50K .......... .......... .......... .......... .......... 16% 2.47M 0s 2024-04-02T12:30:39.963 INFO:teuthology.orchestra.run.smithi129.stderr: 100K .......... .......... .......... .......... .......... 24% 54.0M 0s 2024-04-02T12:30:39.997 INFO:teuthology.orchestra.run.smithi129.stderr: 150K .......... .......... .......... .......... .......... 32% 1.44M 0s 2024-04-02T12:30:39.997 INFO:teuthology.orchestra.run.smithi129.stderr: 200K .......... .......... .......... .......... .......... 41% 123M 0s 2024-04-02T12:30:39.997 INFO:teuthology.orchestra.run.smithi129.stderr: 250K .......... .......... .......... .......... .......... 49% 76.9M 0s 2024-04-02T12:30:40.009 INFO:teuthology.orchestra.run.smithi129.stderr: 300K .......... .......... .......... .......... .......... 57% 4.17M 0s 2024-04-02T12:30:40.020 INFO:teuthology.orchestra.run.smithi129.stderr: 350K .......... .......... .......... .......... .......... 65% 5.04M 0s 2024-04-02T12:30:40.020 INFO:teuthology.orchestra.run.smithi129.stderr: 400K .......... .......... .......... .......... .......... 74% 119M 0s 2024-04-02T12:30:40.020 INFO:teuthology.orchestra.run.smithi129.stderr: 450K .......... .......... .......... .......... .......... 82% 45.6M 0s 2024-04-02T12:30:40.029 INFO:teuthology.orchestra.run.smithi129.stderr: 500K .......... .......... .......... .......... .......... 90% 4.79M 0s 2024-04-02T12:30:40.031 INFO:teuthology.orchestra.run.smithi129.stderr: 550K .......... .......... .......... .......... .......... 98% 60.4M 0s 2024-04-02T12:30:40.031 INFO:teuthology.orchestra.run.smithi129.stderr: 600K ...... 100% 11.4T=0.1s 2024-04-02T12:30:40.031 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:40.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02 12:30:40 (4.11 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-02T12:30:40.031 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/ 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/collectl.conf 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/ 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Socket.html 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/colmux.html 2024-04-02T12:30:40.063 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Import.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/style.css 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Process.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-02T12:30:40.064 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Export.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Startup.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Performance.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Messages.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Logging.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Misc.html 2024-04-02T12:30:40.065 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Examples.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Data.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Playback.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Hello.html 2024-04-02T12:30:40.066 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Network.html 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Memory.html 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/Disks.html 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/service/ 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/service/collectl.service 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/initd/ 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/initd/collectl 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/misc.ph 2024-04-02T12:30:40.067 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/gexpr.ph 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/statsd.ph 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/envrules.std 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/hello.ph 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/vmsum.ph 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/proctree.ph 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/README-WINDOWS 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/formatit.ph 2024-04-02T12:30:40.068 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/colmux 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/INSTALL 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/graphite.ph 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/vmstat.ph 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/README 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/man1/ 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/man1/collectl.1 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/man1/colmux.1 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/lexpr.ph 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/client.pl 2024-04-02T12:30:40.070 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/collectl 2024-04-02T12:30:40.071 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/COPYING 2024-04-02T12:30:40.071 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/vnet.ph 2024-04-02T12:30:40.071 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/GPL 2024-04-02T12:30:40.071 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/UNINSTALL 2024-04-02T12:30:40.071 INFO:teuthology.orchestra.run.smithi129.stdout:collectl-4.3.1/ARTISTIC 2024-04-02T12:30:40.119 INFO:teuthology.orchestra.run.smithi129.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-02T12:30:40.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-02T12:30:40.692 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:03:38 ago on Tue 02 Apr 2024 12:27:02 PM UTC. 2024-04-02T12:30:40.793 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-02T12:30:40.796 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-02T12:30:40.855 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: librbd-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 23 k 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: perf x86_64 5.14.0-434.el9 CentOS-AppStream 2.2 M 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: libradospp-devel x86_64 2:19.0.0-2589.g38caedd8.el9 ceph 30 k 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:40.856 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-02T12:30:40.857 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-04-02T12:30:40.857 INFO:teuthology.orchestra.run.smithi129.stdout:Install 5 Packages 2024-04-02T12:30:40.857 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:40.857 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.3 M 2024-04-02T12:30:40.857 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 11 M 2024-04-02T12:30:40.857 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-02T12:30:41.259 INFO:teuthology.orchestra.run.smithi129.stdout:(1/5): perf-5.14.0-434.el9.x86_64.rpm 6.6 MB/s | 2.2 MB 00:00 2024-04-02T12:30:41.285 INFO:teuthology.orchestra.run.smithi129.stdout:(2/5): libradospp-devel-19.0.0-2589.g38caedd8.e 87 kB/s | 30 kB 00:00 2024-04-02T12:30:41.309 INFO:teuthology.orchestra.run.smithi129.stdout:(3/5): librbd-devel-19.0.0-2589.g38caedd8.el9.x 62 kB/s | 23 kB 00:00 2024-04-02T12:30:41.643 INFO:teuthology.orchestra.run.smithi129.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 39 kB/s | 13 kB 00:00 2024-04-02T12:30:41.918 INFO:teuthology.orchestra.run.smithi129.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 180 kB/s | 118 kB 00:00 2024-04-02T12:30:41.922 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-02T12:30:41.923 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.2 MB/s | 2.3 MB 00:01 2024-04-02T12:30:41.923 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-02T12:30:41.941 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-02T12:30:41.941 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-02T12:30:42.052 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-02T12:30:42.054 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-02T12:30:42.328 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-02T12:30:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-02T12:30:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-02T12:30:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradospp-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 3/5 2024-04-02T12:30:42.779 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librbd-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 4/5 2024-04-02T12:30:42.859 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : perf-5.14.0-434.el9.x86_64 5/5 2024-04-02T12:30:43.663 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: perf-5.14.0-434.el9.x86_64 5/5 2024-04-02T12:30:43.663 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradospp-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 1/5 2024-04-02T12:30:43.663 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2/5 2024-04-02T12:30:43.664 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : perf-5.14.0-434.el9.x86_64 3/5 2024-04-02T12:30:43.664 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-02T12:30:44.000 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: libradospp-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: librbd-devel-2:19.0.0-2589.g38caedd8.el9.x86_64 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: pdsh-2.34-7.el9.x86_64 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: perf-5.14.0-434.el9.x86_64 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-02T12:30:44.001 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-02T12:30:44.313 INFO:tasks.cbt:benchmark: radosbench 2024-04-02T12:30:44.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-02T12:30:44.342 DEBUG:teuthology.orchestra.run.smithi129:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - Settings.cluster: 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: 'clients': ['ubuntu@smithi129.front.sepia.ceph.com'], 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: 'head': 'ubuntu@smithi129.front.sepia.ceph.com', 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: 'is_teuthology': True, 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: 'iterations': 1, 2024-04-02T12:30:44.719 INFO:teuthology.orchestra.run.smithi129.stderr: 'mons': ['ubuntu@smithi129.front.sepia.ceph.com'], 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'osds': ['ubuntu@smithi129.front.sepia.ceph.com'], 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'osds_per_node': 3, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'pid_dir': '/var/run/ceph', 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'pgp_size': 256, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'replication': 'replicated'}}, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'rebuild_every_test': False, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'rgws': {'ubuntu@smithi129.front.sepia.ceph.com': None}, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'use_existing': True, 2024-04-02T12:30:44.720 INFO:teuthology.orchestra.run.smithi129.stderr: 'user': 'ubuntu'} 2024-04-02T12:30:44.740 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-04-02T12:30:44.741 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:30:44.741 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:30:44.741 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2024-04-02T12:30:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:30:44.744 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:30:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:30:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:30:44.746 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2024-04-02T12:30:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:30:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:30:44.749 INFO:teuthology.orchestra.run.smithi129.stderr:12:30:44 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:31:44.792 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.792 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:31:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.875 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:31:44.902 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.902 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-04-02T12:31:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.904 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-04-02T12:31:44.906 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.906 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.906 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2024-04-02T12:31:44.923 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.923 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/$(basename ${i}).smithi129.front.sepia.ceph.com ; done 2024-04-02T12:31:44.927 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.927 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.928 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-04-02T12:31:44.945 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - WARNING - cbt - sudo killall -9 rados 2024-04-02T12:31:44.946 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - WARNING - cbt - error 1 seen, continuing anyway... 2024-04-02T12:31:44.965 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2024-04-02T12:31:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-04-02T12:31:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.966 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-04-02T12:31:44.982 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.982 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.982 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2024-04-02T12:31:44.984 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.984 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:31:44.986 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.986 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.986 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2024-04-02T12:31:44.989 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:44 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.989 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:31:44.989 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:31:44.989 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.989 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:44.990 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-04-02T12:31:45.300 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:45.300 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:45.300 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-04-02T12:31:46.216 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:46 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:46.216 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:46 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:46.216 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-04-02T12:31:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:47 - INFO - cbt - Checking Health after pool creation. 2024-04-02T12:31:52.226 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:52 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:52.226 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:52 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:52.226 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-02T12:31:52.579 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:52 - INFO - cbt - Final Pool Health Check. 2024-04-02T12:31:57.585 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.585 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.585 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-02T12:31:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.912 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:31:57.923 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.923 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:31:57.945 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.945 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-04-02T12:31:57.947 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.947 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-04-02T12:31:57.949 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.949 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:57.949 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-02T12:31:58.458 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:58 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:58.458 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-02T12:31:59.040 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:59.040 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:59.040 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-04-02T12:31:59.407 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:59.407 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:59.407 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2024-04-02T12:31:59.447 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:31:59.447 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:31:59.447 INFO:teuthology.orchestra.run.smithi129.stderr:12:31:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2024-04-02T12:32:00.181 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.181 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.181 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2024-04-02T12:32:00.183 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.184 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T12:32:00.185 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.185 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.185 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2024-04-02T12:32:00.188 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.188 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T12:32:00.188 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T12:32:00.188 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - INFO - cbt - Running radosbench write test. 2024-04-02T12:32:00.188 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.188 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.189 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2024-04-02T12:32:00.190 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.190 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:32:00.190 INFO:teuthology.orchestra.run.smithi129.stderr:12:32:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2024-04-02T12:33:01.309 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.310 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T12:33:01.403 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.403 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T12:33:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.426 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2024-04-02T12:33:01.428 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.428 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2024-04-02T12:33:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.429 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.430 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2024-04-02T12:33:01.739 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.752 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.752 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2024-04-02T12:33:01.836 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - get_fqdn_local()=smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.836 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/$(basename ${i}).smithi129.front.sepia.ceph.com ; done 2024-04-02T12:33:01.873 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - INFO - cbt - Convert results to json format. 2024-04-02T12:33:01.873 INFO:teuthology.orchestra.run.smithi129.stderr:12:33:01 - DEBUG - cbt - Nodes : ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:01.894 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-02T12:33:01.904 DEBUG:teuthology.orchestra.run.smithi129:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-02T12:33:02.150 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-02T12:33:02.150 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-02T12:33:02.150 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-02T12:33:02.264 INFO:teuthology.orchestra.run.smithi129.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi129.front.sepia.ceph.com 2024-04-02T12:33:02.264 INFO:teuthology.orchestra.run.smithi129.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi129.front.sepia.ceph.com 2024-04-02T12:33:02.265 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:33:02.265 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi129.front.sepia.ceph.com of=/dev/stdout 2024-04-02T12:33:02.283 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T12:33:02.283 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2024-04-02T12:33:02.339 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T12:33:02.339 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T12:33:02.339 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-02T12:33:02.339 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi129.front.sepia.ceph.com of=/dev/stdout 2024-04-02T12:33:02.394 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T12:33:02.394 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2024-04-02T12:33:02.449 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T12:33:02.449 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T12:33:02.458 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T12:33:02.465 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T12:33:02.465 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-02T12:33:02.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-02T12:33:02.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:33:02.634+0000 7f5a0d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:33:02.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-02T12:33:02.642+0000 7f5a0d35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T12:33:02.845 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-02T12:33:02.846 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-02T12:33:02.846 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-02T12:33:02.846 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:33:03.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-02T12:33:08.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-02T12:33:08.947 INFO:tasks.ceph.osd.0:Stopped 2024-04-02T12:33:08.948 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-02T12:33:08.948 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:33:09.032 INFO:tasks.ceph.osd.1:Stopped 2024-04-02T12:33:09.032 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-02T12:33:09.032 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:33:09.116 INFO:tasks.ceph.osd.2:Stopped 2024-04-02T12:33:09.116 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-02T12:33:09.117 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-02T12:33:09.117 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:33:09.143 INFO:tasks.ceph.mgr.x:Stopped 2024-04-02T12:33:09.143 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-02T12:33:09.143 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-02T12:33:09.143 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T12:33:09.193 INFO:tasks.ceph.mon.a:Stopped 2024-04-02T12:33:09.193 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-02T12:33:09.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-02T12:33:09.319 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:09.320 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-02T12:33:09.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:09.634 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-02T12:33:09.773 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:09.773 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-02T12:33:09.832 INFO:tasks.ceph:Archiving mon data... 2024-04-02T12:33:09.833 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527/data/mon.a.tgz 2024-04-02T12:33:09.833 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-04-02T12:33:09.854 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.kvEqYprAAx 2024-04-02T12:33:09.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kvEqYprAAx 2024-04-02T12:33:09.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.kvEqYprAAx 2024-04-02T12:33:10.076 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.kvEqYprAAx is 181KB 2024-04-02T12:33:10.131 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.kvEqYprAAx 2024-04-02T12:33:10.146 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-02T12:33:10.146 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T12:33:10.284 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-02T12:33:10.284 INFO:tasks.ceph:Archiving crash dumps... 2024-04-02T12:33:10.286 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527/remote/smithi129/crash 2024-04-02T12:33:10.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T12:33:10.351 INFO:tasks.ceph:Compressing logs... 2024-04-02T12:33:10.351 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T12:33:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-02T12:33:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31098.log 2024-04-02T12:33:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31728.log 2024-04-02T12:33:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.31098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32516.log 2024-04-02T12:33:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.31728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32627.log 2024-04-02T12:33:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.32516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30886.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.32627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29776.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.32627.log.gz 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31770.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30659.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.31770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30762.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29886.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30561.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.29886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31496.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30561.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28316.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31139.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31602.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30124.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29688.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32414.log 2024-04-02T12:33:10.448 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31370.log 2024-04-02T12:33:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32865.log 2024-04-02T12:33:10.449 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31224.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32563.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32540.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31265.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31622.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32628.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30720.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31011.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32479.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-02T12:33:10.457 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29926.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31350.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31244.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31118.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31476.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31396.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30154.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30885.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30884.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30617.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30972.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29966.log 2024-04-02T12:33:10.458 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30804.log 2024-04-02T12:33:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.tmp-client.admin.28316.log: /var/log/ceph/ceph-client.admin.31496.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31686.log 2024-04-02T12:33:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.30885.log: /var/log/ceph/ceph-client.admin.30617.log: /var/log/ceph/ceph-client.admin.32865.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.30884.log: /var/log/ceph/ceph-client.admin.30972.log: /var/log/ceph/ceph-client.admin.32628.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.0.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.32865.log.gz 2024-04-02T12:33:10.459 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-04-02T12:33:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.31602.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.32628.log.gz 2024-04-02T12:33:10.459 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.31139.log: /var/log/ceph/ceph-client.admin.31622.log: /var/log/ceph/ceph-client.admin.31686.log: /var/log/ceph/ceph-client.admin.31370.log: /var/log/ceph/ceph-client.admin.30124.log: /var/log/ceph/ceph-client.admin.29688.log: /var/log/ceph/ceph-client.admin.32414.log: /var/log/ceph/ceph-client.admin.31224.log: /var/log/ceph/ceph-client.admin.32563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31518.log 2024-04-02T12:33:10.463 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.31265.log: /var/log/ceph/ceph-client.admin.30720.log: /var/log/ceph/ceph-client.admin.32540.log: /var/log/ceph/ceph-client.admin.30804.log: /var/log/ceph/ceph-client.admin.29966.log: /var/log/ceph/ceph-client.admin.31011.log: /var/log/ceph/ceph-client.admin.32479.log: /var/log/ceph/ceph-client.admin.29926.log: /var/log/ceph/ceph-client.admin.31350.log: /var/log/ceph/ceph-client.admin.31244.log: /var/log/ceph/ceph-client.admin.31118.log: /var/log/ceph/ceph-client.admin.31396.log: /var/log/ceph/ceph-client.admin.31476.log: /var/log/ceph/ceph-client.admin.30154.log: /var/log/ceph/ceph-client.admin.31012.log: /var/log/ceph/ceph-client.admin.31518.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30762.log.gz 2024-04-02T12:33:10.463 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30884.log.gz 2024-04-02T12:33:10.464 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30885.log.gz 2024-04-02T12:33:10.464 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30886.log.gz 2024-04-02T12:33:10.464 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30972.log.gz 2024-04-02T12:33:10.464 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31098.log.gz 2024-04-02T12:33:10.465 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31139.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.31350.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% 47.6% -- replaced with /var/log/ceph/ceph-client.admin.31265.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31244.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31224.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30804.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.31011.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31118.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31370.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31396.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31476.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31496.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31518.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31602.log.gz 2024-04-02T12:33:10.467 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31622.log.gz 2024-04-02T12:33:10.468 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.31686.log.gz 2024-04-02T12:33:10.468 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31728.log.gz 2024-04-02T12:33:10.468 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31770.log.gz 2024-04-02T12:33:10.506 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30154.log.gz 2024-04-02T12:33:10.507 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32414.log.gz 2024-04-02T12:33:10.508 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32479.log.gz 2024-04-02T12:33:10.508 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32516.log.gz 2024-04-02T12:33:10.508 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32540.log.gz 2024-04-02T12:33:10.508 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32563.log.gz 2024-04-02T12:33:10.511 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30561.log.gz 2024-04-02T12:33:10.511 INFO:teuthology.orchestra.run.smithi129.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30617.log.gz 2024-04-02T12:33:10.511 INFO:teuthology.orchestra.run.smithi129.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30659.log.gz 2024-04-02T12:33:10.511 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30720.log.gz 2024-04-02T12:33:10.514 INFO:teuthology.orchestra.run.smithi129.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28316.log.gz 2024-04-02T12:33:10.523 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29776.log.gz 2024-04-02T12:33:10.523 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 50.4% 2024-04-02T12:33:10.523 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29688.log.gz 2024-04-02T12:33:10.523 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-mon.a.log.gz 84.8% 2024-04-02T12:33:10.523 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-02T12:33:10.531 INFO:teuthology.orchestra.run.smithi129.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29886.log.gz 2024-04-02T12:33:10.532 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29926.log.gz 2024-04-02T12:33:10.532 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29966.log.gz 2024-04-02T12:33:10.532 INFO:teuthology.orchestra.run.smithi129.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30124.log.gz 2024-04-02T12:33:10.532 INFO:teuthology.orchestra.run.smithi129.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-02T12:33:11.726 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-02T12:33:11.869 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-02T12:33:11.946 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-02T12:33:11.948 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-02T12:33:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m1.543s 2024-04-02T12:33:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m2.725s 2024-04-02T12:33:11.948 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.155s 2024-04-02T12:33:11.948 INFO:tasks.ceph:Archiving logs... 2024-04-02T12:33:11.949 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527/remote/smithi129/log 2024-04-02T12:33:11.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T12:33:12.481 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T12:33:12.491 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T12:33:12.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T12:33:12.545 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T12:33:12.554 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T12:33:12.554 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T12:33:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-04-02T12:33:12.640 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T12:33:12.640 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-04-02T12:33:12.640 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +101us[ +136us] +/- 74ms 2024-04-02T12:33:12.640 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -1023us[ -989us] +/- 60ms 2024-04-02T12:33:12.640 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +1396us[+1430us] +/- 84ms 2024-04-02T12:33:12.640 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T12:33:12.648 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T12:33:12.656 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T12:33:12.657 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T12:33:12.666 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T12:33:12.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T12:33:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:33:12.852 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T12:33:12.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T12:33:12.870 INFO:teuthology.task.internal:Duration was 781.350678 seconds 2024-04-02T12:33:12.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T12:33:12.879 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T12:33:12.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T12:33:13.023 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T12:33:13.135 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T12:33:13.135 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-04-02T12:33:13.135 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T12:33:13.155 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T12:33:13.155 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T12:33:13.223 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T12:33:13.223 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T12:33:13.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T12:33:13.376 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T12:33:13.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T12:33:13.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T12:33:13.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T12:33:13.470 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-04-02T12:33:13.495 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T12:33:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T12:33:13.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T12:33:13.558 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T12:33:13.558 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-04-02_11:03:04-crimson-rados-wip-matanb-crimson-testing-apr-2-distro-crimson-smithi/7635527/remote/smithi129 2024-04-02T12:33:13.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T12:33:13.650 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T12:33:13.651 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T12:33:13.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T12:33:13.716 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T12:33:13.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T12:33:13.725 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T12:33:13.725 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T12:33:13.758 INFO:teuthology.orchestra.run.smithi129.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 12:33 /home/ubuntu/cephtest 2024-04-02T12:33:13.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-02T12:33:13.759 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T12:33:13.770 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T12:33:13.792 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-testing-apr-2&var-job_id_selected=7635527 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} duration: 781.350677728653 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-04-02T12:33:13.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T12:33:13.886 INFO:teuthology.run:pass