2023-12-20T10:41:54.540 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-20T10:41:54.540 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823 --name matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} -- /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823/orig.config.yaml 2023-12-20T10:41:54.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-20T10:41:54.667 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823 branch: wip-matanb-crimson-only-submit-error-cleanup description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: null first_in_suite: false job_id: '7497823' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-only-submit-error-cleanup ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: a8ad38c3a13ec7dc3249f402106905f8db892f83 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: a8ad38c3a13ec7dc3249f402106905f8db892f83 workunit: branch: wip-matanb-crimson-testing-misc env: CRIMSON_COMPAT: '1' sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 241 sha1: a8ad38c3a13ec7dc3249f402106905f8db892f83 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-testing-misc suite_path: /home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa suite_relpath: qa suite_repo: https://github.com/Matan-B/ceph.git suite_sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbLJJMp69XYhsm0XJoKroTJ3395FUcOJ4LWihlPp5H9DghW8UMjZ36ohYXosJASFQtlHNVz8rZw1l2vGy5X6Zg= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-20_10:24:11 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-20T10:41:54.667 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa; will attempt to use it 2023-12-20T10:41:54.669 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa/tasks 2023-12-20T10:41:54.669 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-20T10:41:54.671 INFO:teuthology.task.internal:Checking packages... 2023-12-20T10:41:54.695 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'a8ad38c3a13ec7dc3249f402106905f8db892f83' 2023-12-20T10:41:54.696 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-20T10:41:54.696 INFO:teuthology.packaging:ref: None 2023-12-20T10:41:54.696 INFO:teuthology.packaging:tag: None 2023-12-20T10:41:54.696 INFO:teuthology.packaging:branch: wip-matanb-crimson-only-submit-error-cleanup 2023-12-20T10:41:54.696 INFO:teuthology.packaging:sha1: a8ad38c3a13ec7dc3249f402106905f8db892f83 2023-12-20T10:41:54.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-only-submit-error-cleanup 2023-12-20T10:41:54.930 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-282.ga8ad38c3 2023-12-20T10:41:54.931 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-20T10:41:54.939 INFO:teuthology.task.internal:no buildpackages task found 2023-12-20T10:41:54.939 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-20T10:41:54.987 INFO:teuthology.task.internal:Saving configuration 2023-12-20T10:41:55.004 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-20T10:41:55.011 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-20T10:41:55.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-20 10:37:07.402794', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbLJJMp69XYhsm0XJoKroTJ3395FUcOJ4LWihlPp5H9DghW8UMjZ36ohYXosJASFQtlHNVz8rZw1l2vGy5X6Zg='} 2023-12-20T10:41:55.036 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-20T10:41:55.042 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-20T10:41:55.043 INFO:teuthology.run_tasks:Running task console_log... 2023-12-20T10:41:55.094 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f85d6be3dc0>, signals=[15]) 2023-12-20T10:41:55.095 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-20T10:41:55.101 INFO:teuthology.task.internal:Opening connections... 2023-12-20T10:41:55.101 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:41:55.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-20T10:41:55.203 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-20T10:41:55.210 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2023-12-20T10:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2023-12-20T10:41:55.275 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2023-12-20T10:41:55.332 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2023-12-20T10:41:55.332 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-20T10:41:55.333 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-20T10:41:55.334 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-12-20T10:41:55.357 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-20T10:41:55.365 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-20T10:41:55.379 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-20T10:41:55.379 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2023-12-20T10:41:55.397 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-20T10:41:55.403 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-20T10:41:55.403 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2023-12-20T10:41:55.466 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-20T10:41:55.497 INFO:teuthology.run_tasks:Running task kernel... 2023-12-20T10:41:55.511 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-20T10:41:55.511 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-20T10:41:55.511 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-20T10:41:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-20T10:41:55.533 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-12-20T10:41:55.590 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2023-12-20T10:41:55.591 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2023-12-20T10:41:55.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2023-12-20T10:41:57.670 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-20T10:41:59.036 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-12-20T10:42:00.352 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 7.9 MB 00:00 2023-12-20T10:42:05.120 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 18 MB 00:03 2023-12-20T10:42:09.679 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 5.8 MB 00:01 2023-12-20T10:42:11.313 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-12-20T10:42:12.398 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2023-12-20T10:42:17.273 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-20T10:42:19.521 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-20T10:42:19.521 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-20T10:42:19.615 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-20T10:42:19.621 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 96 M 2023-12-20T10:42:19.622 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 121 M 2023-12-20T10:42:19.623 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:42:20.424 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 8.2 MB/s | 5.6 MB 00:00 2023-12-20T10:42:20.916 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 17 MB/s | 20 MB 00:01 2023-12-20T10:42:21.508 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 21 MB/s | 38 MB 00:01 2023-12-20T10:42:21.825 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 23 MB/s | 32 MB 00:01 2023-12-20T10:42:21.825 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:42:21.825 INFO:teuthology.orchestra.run.smithi006.stdout:Total 44 MB/s | 96 MB 00:02 2023-12-20T10:42:22.720 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:42:23.162 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:42:23.162 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:42:24.514 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:42:24.515 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:42:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:42:27.971 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-20T10:42:28.100 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-20T10:42:31.003 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-20T10:42:31.178 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-20T10:42:36.171 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-20T10:42:36.313 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-20T10:42:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-20T10:43:12.185 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-20T10:43:12.230 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-20T10:43:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-20T10:43:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-20T10:43:12.730 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-20T10:43:12.731 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-20T10:43:13.150 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-20T10:43:13.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:43:13.150 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:43:13.151 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-20T10:43:13.151 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-20T10:43:13.151 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-20T10:43:13.151 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-20T10:43:13.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:43:13.151 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:43:13.664 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2023-12-20T10:43:13.812 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-20T10:43:13.812 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-20T10:43:13.812 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-20T10:43:13.812 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-20T10:43:13.813 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-20T10:43:13.813 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-20T10:43:13.813 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2023-12-20T10:43:13.813 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2023-12-20T10:43:13.935 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-20T10:43:13.936 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-20T10:43:13.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2023-12-20T10:43:13.981 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:43:13.982 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2023-12-20T10:43:13.983 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-20T10:43:13.983 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-391.el9.x86_64 2023-12-20T10:43:13.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-20T10:43:14.575 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2023-12-20T10:43:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-20T10:43:16.159 INFO:teuthology.orchestra.run.smithi006.stderr:done 2023-12-20T10:43:16.163 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-12-20T10:43:16.219 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.mDMAnunVBp 2023-12-20T10:43:16.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mDMAnunVBp 2023-12-20T10:43:16.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.mDMAnunVBp 2023-12-20T10:43:16.451 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.mDMAnunVBp is 6KB 2023-12-20T10:43:16.498 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.mDMAnunVBp 2023-12-20T10:43:16.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2023-12-20T10:43:16.591 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-20T10:43:16.591 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-20T10:43:16.591 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-20T10:43:16.591 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-20T10:43:16.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-20T10:43:16.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2023-12-20T10:43:46.786 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-20T10:43:46.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-20T10:43:46.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-20T10:44:05.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-12-20T10:44:14.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-20T10:44:14.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-20T10:45:14.338 DEBUG:teuthology.orchestra.remote:timed out 2023-12-20T10:45:26.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-20T10:45:26.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-20T10:45:26.642 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-12-20T10:45:27.042 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-20T10:45:27.042 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-20T10:45:27.042 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-12-20T10:45:27.058 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-391.el9.x86_64 2023-12-20T10:45:27.059 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-20T10:45:27.059 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-20T10:45:27.059 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-20T10:45:28.059 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-20T10:45:28.060 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-20T10:45:28.134 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2023-12-20T10:45:28.175 DEBUG:teuthology.parallel:result is None 2023-12-20T10:45:28.176 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-20T10:45:28.183 INFO:teuthology.task.internal:Creating test directory... 2023-12-20T10:45:28.183 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-20T10:45:28.202 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-20T10:45:28.213 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-20T10:45:28.244 INFO:teuthology.task.internal:Creating archive directory... 2023-12-20T10:45:28.244 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-20T10:45:28.279 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-20T10:45:28.289 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-20T10:45:28.290 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-20T10:45:28.354 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-20T10:45:28.370 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-20T10:45:28.372 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-20T10:45:28.379 INFO:teuthology.task.internal:Configuring sudo... 2023-12-20T10:45:28.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-20T10:45:28.451 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-20T10:45:28.466 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-20T10:45:28.466 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-20T10:45:28.511 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-20T10:45:28.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-20T10:45:28.654 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-20T10:45:28.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-20T10:45:28.790 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:45:28.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-20T10:45:28.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2023-12-20T10:45:28.954 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-20T10:45:29.208 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-20T10:45:29.216 INFO:teuthology.task.internal:Starting timer... 2023-12-20T10:45:29.216 INFO:teuthology.run_tasks:Running task pcp... 2023-12-20T10:45:29.256 INFO:teuthology.run_tasks:Running task selinux... 2023-12-20T10:45:29.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2023-12-20T10:45:29.339 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2023-12-20T10:45:29.341 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-20T10:45:29.341 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2023-12-20T10:45:29.379 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2023-12-20T10:45:29.380 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive'} 2023-12-20T10:45:29.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-20T10:45:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-20T10:45:29.455 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-20T10:45:29.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2023-12-20T10:45:29.539 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-20T10:45:29.549 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-20T10:45:29.555 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-20T10:45:29.706 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-20T10:45:29.724 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-20T10:45:29.726 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-20T10:51:22.969 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com')] 2023-12-20T10:51:22.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-20T10:51:22.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-20T10:51:23.076 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-12-20T10:51:23.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-20T10:51:23.145 INFO:teuthology.run_tasks:Running task clock... 2023-12-20T10:51:23.157 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-20T10:51:23.157 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-20T10:51:23.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-20T10:51:23.221 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-20T10:51:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-20T10:51:23.278 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2023-12-20T10:51:23.295 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2023-12-20T10:51:23.315 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-20T10:51:23.335 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-20T10:51:23.383 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2023-12-20T10:51:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-20T10:51:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2023-12-20T10:51:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-20T10:51:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-20T10:51:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-20T10:51:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-20T10:51:23.387 INFO:teuthology.run_tasks:Running task install... 2023-12-20T10:51:23.396 DEBUG:teuthology.task.install:project ceph 2023-12-20T10:51:23.396 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83'}} 2023-12-20T10:51:23.396 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83'} 2023-12-20T10:51:23.396 INFO:teuthology.task.install:Using flavor: crimson 2023-12-20T10:51:23.404 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-20T10:51:23.405 INFO:teuthology.task.install:extra packages: [] 2023-12-20T10:51:23.405 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83', 'tag': None, 'wait_for_package': False} 2023-12-20T10:51:23.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a8ad38c3a13ec7dc3249f402106905f8db892f83 2023-12-20T10:51:23.623 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-only-submit-error-cleanup/a8ad38c3a13ec7dc3249f402106905f8db892f83/centos/9/flavors/crimson/ 2023-12-20T10:51:23.623 INFO:teuthology.task.install.rpm:Package version is 19.0.0-282.ga8ad38c3 2023-12-20T10:51:23.897 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-only-submit-error-cleanup/a8ad38c3a13ec7dc3249f402106905f8db892f83/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-only-submit-error-cleanup/a8ad38c3a13ec7dc3249f402106905f8db892f83/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-only-submit-error-cleanup/a8ad38c3a13ec7dc3249f402106905f8db892f83/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-20T10:51:23.897 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:51:23.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-20T10:51:23.932 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-20T10:51:23.933 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a8ad38c3a13ec7dc3249f402106905f8db892f83/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-20T10:51:24.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-20T10:51:24.099 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-20T10:51:24.129 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2023-12-20T10:51:24.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2023-12-20T10:51:24.383 INFO:teuthology.orchestra.run.smithi006.stdout:73 files removed 2023-12-20T10:51:24.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2023-12-20T10:51:25.375 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 132 kB/s | 83 kB 00:00 2023-12-20T10:51:26.061 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 15 kB/s | 9.5 kB 00:00 2023-12-20T10:51:26.594 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2023-12-20T10:51:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-20T10:51:28.663 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2023-12-20T10:51:31.087 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 7.9 MB 00:02 2023-12-20T10:51:34.030 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 18 MB 00:01 2023-12-20T10:51:39.039 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.8 MB 00:01 2023-12-20T10:51:40.515 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2023-12-20T10:51:42.386 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-20T10:51:47.292 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-20T10:51:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:51:49.914 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:51:49.914 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:51:49.914 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:51:49.914 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:51:49.914 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 8.3 M 2023-12-20T10:51:49.914 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 3.1 M 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 3.1 M 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 6.6 M 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 17 M 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 24 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 751 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-20T10:51:49.915 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 498 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 4.5 M 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 45 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 130 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 161 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 320 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 298 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 99 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-20T10:51:49.916 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout:Install 27 Packages 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 54 M 2023-12-20T10:51:49.917 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:51:52.701 INFO:teuthology.orchestra.run.smithi006.stdout:(1/29): ceph-base-19.0.0-282.ga8ad38c3.el9.x86_ 2.7 MB/s | 6.6 MB 00:02 2023-12-20T10:51:52.776 INFO:teuthology.orchestra.run.smithi006.stdout:(2/29): ceph-selinux-19.0.0-282.ga8ad38c3.el9.x 322 kB/s | 24 kB 00:00 2023-12-20T10:51:53.001 INFO:teuthology.orchestra.run.smithi006.stdout:(3/29): libcephfs2-19.0.0-282.ga8ad38c3.el9.x86 3.3 MB/s | 751 kB 00:00 2023-12-20T10:51:53.160 INFO:teuthology.orchestra.run.smithi006.stdout:(4/29): libradosstriper1-19.0.0-282.ga8ad38c3.e 3.1 MB/s | 498 kB 00:00 2023-12-20T10:51:53.310 INFO:teuthology.orchestra.run.smithi006.stdout:(5/29): ceph-radosgw-19.0.0-282.ga8ad38c3.el9.x 2.8 MB/s | 8.3 MB 00:03 2023-12-20T10:51:53.394 INFO:teuthology.orchestra.run.smithi006.stdout:(6/29): python3-ceph-argparse-19.0.0-282.ga8ad3 543 kB/s | 45 kB 00:00 2023-12-20T10:51:53.469 INFO:teuthology.orchestra.run.smithi006.stdout:(7/29): python3-ceph-common-19.0.0-282.ga8ad38c 1.7 MB/s | 130 kB 00:00 2023-12-20T10:51:53.551 INFO:teuthology.orchestra.run.smithi006.stdout:(8/29): python3-cephfs-19.0.0-282.ga8ad38c3.el9 1.9 MB/s | 161 kB 00:00 2023-12-20T10:51:53.643 INFO:teuthology.orchestra.run.smithi006.stdout:(9/29): python3-rados-19.0.0-282.ga8ad38c3.el9. 3.4 MB/s | 320 kB 00:00 2023-12-20T10:51:53.727 INFO:teuthology.orchestra.run.smithi006.stdout:(10/29): python3-rbd-19.0.0-282.ga8ad38c3.el9.x 3.5 MB/s | 298 kB 00:00 2023-12-20T10:51:53.785 INFO:teuthology.orchestra.run.smithi006.stdout:(11/29): python3-rgw-19.0.0-282.ga8ad38c3.el9.x 1.7 MB/s | 99 kB 00:00 2023-12-20T10:51:53.928 INFO:teuthology.orchestra.run.smithi006.stdout:(12/29): librgw2-19.0.0-282.ga8ad38c3.el9.x86_6 5.8 MB/s | 4.5 MB 00:00 2023-12-20T10:51:53.953 INFO:teuthology.orchestra.run.smithi006.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 106 kB 00:00 2023-12-20T10:51:53.978 INFO:teuthology.orchestra.run.smithi006.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-20T10:51:54.003 INFO:teuthology.orchestra.run.smithi006.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2023-12-20T10:51:54.045 INFO:teuthology.orchestra.run.smithi006.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-12-20T10:51:54.071 INFO:teuthology.orchestra.run.smithi006.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-12-20T10:51:54.096 INFO:teuthology.orchestra.run.smithi006.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2023-12-20T10:51:54.271 INFO:teuthology.orchestra.run.smithi006.stdout:(19/29): ceph-common-19.0.0-282.ga8ad38c3.el9.x 4.4 MB/s | 17 MB 00:03 2023-12-20T10:51:54.521 INFO:teuthology.orchestra.run.smithi006.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2023-12-20T10:51:54.697 INFO:teuthology.orchestra.run.smithi006.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2023-12-20T10:51:54.730 INFO:teuthology.orchestra.run.smithi006.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 469 kB/s | 309 kB 00:00 2023-12-20T10:51:54.797 INFO:teuthology.orchestra.run.smithi006.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2023-12-20T10:51:54.914 INFO:teuthology.orchestra.run.smithi006.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2023-12-20T10:51:54.973 INFO:teuthology.orchestra.run.smithi006.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2023-12-20T10:51:55.123 INFO:teuthology.orchestra.run.smithi006.stdout:(26/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2023-12-20T10:51:55.187 INFO:teuthology.orchestra.run.smithi006.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2023-12-20T10:51:55.473 INFO:teuthology.orchestra.run.smithi006.stdout:(28/29): librados2-19.0.0-282.ga8ad38c3.el9.x86 6.3 MB/s | 3.1 MB 00:00 2023-12-20T10:51:55.590 INFO:teuthology.orchestra.run.smithi006.stdout:(29/29): librbd1-19.0.0-282.ga8ad38c3.el9.x86_6 6.6 MB/s | 3.1 MB 00:00 2023-12-20T10:51:55.594 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:51:55.594 INFO:teuthology.orchestra.run.smithi006.stdout:Total 9.5 MB/s | 54 MB 00:05 2023-12-20T10:51:55.595 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:51:55.636 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:51:55.636 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:51:56.232 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:51:56.232 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:51:57.615 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:51:58.045 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-20T10:51:58.076 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/31 2023-12-20T10:51:58.152 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/31 2023-12-20T10:51:58.244 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-282.ga8ad38c3.el9.x86_64 3/31 2023-12-20T10:51:58.278 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-282.ga8ad38c3.el9.x86_64 4/31 2023-12-20T10:51:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-282.ga8ad38c3.el9.x86_64 4/31 2023-12-20T10:51:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-12-20T10:51:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-12-20T10:51:58.454 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-12-20T10:51:58.497 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-282.ga8ad38c3.el9.x 8/31 2023-12-20T10:51:58.568 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-282.ga8ad38c3.el9.x86_64 9/31 2023-12-20T10:51:58.597 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-282.ga8ad38c3.el9.x86_64 10/31 2023-12-20T10:51:58.686 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-282.ga8ad38c3.el9.x86_64 10/31 2023-12-20T10:51:58.721 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-12-20T10:51:58.832 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-20T10:51:58.874 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-20T10:51:58.907 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-20T10:51:58.939 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-20T10:51:58.968 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-20T10:51:59.001 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-20T10:51:59.347 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-20T10:51:59.372 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-282.ga8ad38c3.el9.x86_64 19/31 2023-12-20T10:51:59.446 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-282.ga8ad38c3.el9.x86_64 19/31 2023-12-20T10:51:59.477 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 20/31 2023-12-20T10:51:59.513 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-20T10:51:59.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-20T10:51:59.991 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-282.ga8ad38c3.el9.x86_64 23/31 2023-12-20T10:52:00.068 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-282.ga8ad38c3.el9.x86_64 23/31 2023-12-20T10:52:00.141 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 24/31 2023-12-20T10:52:00.185 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86 25/31 2023-12-20T10:52:02.381 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86_64 26/31 2023-12-20T10:52:02.415 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86_64 26/31 2023-12-20T10:52:03.214 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86_64 26/31 2023-12-20T10:52:03.260 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-282.ga8ad38c3.el9.x86_64 27/31 2023-12-20T10:52:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-282.ga8ad38c3.el9.x86_64 27/31 2023-12-20T10:52:03.353 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-20T10:52:03.353 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-20T10:52:03.353 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:03.386 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-282.ga8ad38c3.el9.x86_64 28/31 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-282.ga8ad38c3.el9.x86_64 28/31 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2023-12-20T10:52:13.493 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:14.345 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 29/31 2023-12-20T10:52:14.371 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 29/31 2023-12-20T10:52:14.371 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:52:14.371 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-20T10:52:14.371 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-20T10:52:14.371 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-20T10:52:14.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:14.373 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-20T10:52:14.456 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-20T10:52:14.456 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-20T10:52:15.595 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 3/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-282.ga8ad38c3.el9.x86_64 4/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-282.ga8ad38c3.el9.x86_64 5/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-282.ga8ad38c3.el9.x86_64 6/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-282.ga8ad38c3.el9.x86_64 7/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-282.ga8ad38c3.el9.x 8/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86 9/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-282.ga8ad38c3.el9.x86_64 10/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-282.ga8ad38c3.el9.x86_64 11/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 12/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 13/31 2023-12-20T10:52:15.596 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-20T10:52:15.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-20T10:52:15.598 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-282.ga8ad38c3.el9.x86_64 28/31 2023-12-20T10:52:15.598 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-20T10:52:15.598 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-282.ga8ad38c3.el9.x86_64 30/31 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-20T10:52:16.623 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-20T10:52:16.624 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:16.625 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:52:16.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2023-12-20T10:52:17.519 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:30 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:52:17.671 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:52:17.672 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:52:17.672 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:52:17.672 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 121 M 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 158 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:17.673 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:52:17.674 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:52:17.674 INFO:teuthology.orchestra.run.smithi006.stdout:Install 10 Packages 2023-12-20T10:52:17.674 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:17.674 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 124 M 2023-12-20T10:52:17.674 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.1 G 2023-12-20T10:52:17.674 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:52:17.894 INFO:teuthology.orchestra.run.smithi006.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 186 kB 00:00 2023-12-20T10:52:17.986 INFO:teuthology.orchestra.run.smithi006.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 4.5 MB/s | 418 kB 00:00 2023-12-20T10:52:18.037 INFO:teuthology.orchestra.run.smithi006.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 3.7 MB/s | 186 kB 00:00 2023-12-20T10:52:18.087 INFO:teuthology.orchestra.run.smithi006.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-12-20T10:52:18.137 INFO:teuthology.orchestra.run.smithi006.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2023-12-20T10:52:18.171 INFO:teuthology.orchestra.run.smithi006.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-20T10:52:18.205 INFO:teuthology.orchestra.run.smithi006.stdout:(7/10): libcephsqlite-19.0.0-282.ga8ad38c3.el9. 353 kB/s | 158 kB 00:00 2023-12-20T10:52:18.380 INFO:teuthology.orchestra.run.smithi006.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 697 kB/s | 122 kB 00:00 2023-12-20T10:52:18.496 INFO:teuthology.orchestra.run.smithi006.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 3.9 MB/s | 1.3 MB 00:00 2023-12-20T10:52:28.263 INFO:teuthology.orchestra.run.smithi006.stdout:(10/10): ceph-test-19.0.0-282.ga8ad38c3.el9.x86 12 MB/s | 121 MB 00:10 2023-12-20T10:52:28.268 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:52:28.268 INFO:teuthology.orchestra.run.smithi006.stdout:Total 12 MB/s | 124 MB 00:10 2023-12-20T10:52:28.268 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:52:28.293 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:52:28.293 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:52:29.172 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:52:29.172 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:52:30.348 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:52:30.439 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-12-20T10:52:30.471 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-12-20T10:52:30.516 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-12-20T10:52:30.558 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-12-20T10:52:30.588 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-12-20T10:52:30.620 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-12-20T10:52:30.653 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-12-20T10:52:30.690 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-12-20T10:52:30.718 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-282.ga8ad38c3.el9.x86_64 9/10 2023-12-20T10:52:48.385 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-282.ga8ad38c3.el9.x86_64 9/10 2023-12-20T10:52:48.468 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-282.ga8ad38c3.el9.x86_64 10/10 2023-12-20T10:52:56.761 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:19.0.0-282.ga8ad38c3.el9.x86_64 10/10 2023-12-20T10:52:56.761 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/10 2023-12-20T10:52:56.761 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/10 2023-12-20T10:52:56.761 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-12-20T10:52:56.762 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-12-20T10:52:56.762 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-12-20T10:52:56.762 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-12-20T10:52:56.762 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-12-20T10:52:56.762 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-12-20T10:52:56.762 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-12-20T10:52:57.279 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-12-20T10:52:57.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:57.279 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:52:57.279 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:57.280 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:52:57.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2023-12-20T10:52:58.168 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:11 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:52:58.322 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:52:58.324 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-20T10:52:58.324 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 6.5 k 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 2.0 M 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 1.5 M 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 243 k 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 8.9 M 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 118 M 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-20T10:52:58.325 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-20T10:52:58.326 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout:Install 35 Packages 2023-12-20T10:52:58.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:52:58.328 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 134 M 2023-12-20T10:52:58.328 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.4 G 2023-12-20T10:52:58.328 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:52:59.074 INFO:teuthology.orchestra.run.smithi006.stdout:(1/35): ceph-19.0.0-282.ga8ad38c3.el9.x86_64.rp 17 kB/s | 6.5 kB 00:00 2023-12-20T10:52:59.783 INFO:teuthology.orchestra.run.smithi006.stdout:(2/35): ceph-mgr-19.0.0-282.ga8ad38c3.el9.x86_6 1.4 MB/s | 1.5 MB 00:01 2023-12-20T10:52:59.833 INFO:teuthology.orchestra.run.smithi006.stdout:(3/35): ceph-mds-19.0.0-282.ga8ad38c3.el9.x86_6 1.8 MB/s | 2.0 MB 00:01 2023-12-20T10:52:59.950 INFO:teuthology.orchestra.run.smithi006.stdout:(4/35): ceph-mgr-modules-core-19.0.0-282.ga8ad3 2.0 MB/s | 243 kB 00:00 2023-12-20T10:53:00.067 INFO:teuthology.orchestra.run.smithi006.stdout:(5/35): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-20T10:53:00.109 INFO:teuthology.orchestra.run.smithi006.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-20T10:53:00.135 INFO:teuthology.orchestra.run.smithi006.stdout:(7/35): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-12-20T10:53:00.160 INFO:teuthology.orchestra.run.smithi006.stdout:(8/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2023-12-20T10:53:00.185 INFO:teuthology.orchestra.run.smithi006.stdout:(9/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-12-20T10:53:00.227 INFO:teuthology.orchestra.run.smithi006.stdout:(10/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-20T10:53:00.294 INFO:teuthology.orchestra.run.smithi006.stdout:(11/35): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-20T10:53:00.327 INFO:teuthology.orchestra.run.smithi006.stdout:(12/35): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-20T10:53:00.361 INFO:teuthology.orchestra.run.smithi006.stdout:(13/35): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-12-20T10:53:00.394 INFO:teuthology.orchestra.run.smithi006.stdout:(14/35): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-20T10:53:00.428 INFO:teuthology.orchestra.run.smithi006.stdout:(15/35): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-20T10:53:00.704 INFO:teuthology.orchestra.run.smithi006.stdout:(16/35): luarocks-3.9.2-1.el9.noarch.rpm 550 kB/s | 151 kB 00:00 2023-12-20T10:53:00.762 INFO:teuthology.orchestra.run.smithi006.stdout:(17/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 749 kB/s | 43 kB 00:00 2023-12-20T10:53:00.855 INFO:teuthology.orchestra.run.smithi006.stdout:(18/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2023-12-20T10:53:00.979 INFO:teuthology.orchestra.run.smithi006.stdout:(19/35): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2023-12-20T10:53:01.080 INFO:teuthology.orchestra.run.smithi006.stdout:(20/35): ceph-mon-19.0.0-282.ga8ad38c3.el9.x86_ 4.4 MB/s | 8.9 MB 00:02 2023-12-20T10:53:01.096 INFO:teuthology.orchestra.run.smithi006.stdout:(21/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2023-12-20T10:53:01.130 INFO:teuthology.orchestra.run.smithi006.stdout:(22/35): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2023-12-20T10:53:01.206 INFO:teuthology.orchestra.run.smithi006.stdout:(23/35): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2023-12-20T10:53:01.256 INFO:teuthology.orchestra.run.smithi006.stdout:(24/35): python3-jaraco-text-3.2.0-6.el9.noarch 404 kB/s | 20 kB 00:00 2023-12-20T10:53:01.281 INFO:teuthology.orchestra.run.smithi006.stdout:(25/35): python3-jaraco-collections-3.0.0-8.el9 126 kB/s | 23 kB 00:00 2023-12-20T10:53:01.306 INFO:teuthology.orchestra.run.smithi006.stdout:(26/35): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2023-12-20T10:53:01.365 INFO:teuthology.orchestra.run.smithi006.stdout:(27/35): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2023-12-20T10:53:01.399 INFO:teuthology.orchestra.run.smithi006.stdout:(28/35): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2023-12-20T10:53:01.424 INFO:teuthology.orchestra.run.smithi006.stdout:(29/35): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2023-12-20T10:53:01.457 INFO:teuthology.orchestra.run.smithi006.stdout:(30/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-12-20T10:53:01.483 INFO:teuthology.orchestra.run.smithi006.stdout:(31/35): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2023-12-20T10:53:01.550 INFO:teuthology.orchestra.run.smithi006.stdout:(32/35): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2023-12-20T10:53:01.617 INFO:teuthology.orchestra.run.smithi006.stdout:(33/35): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2023-12-20T10:53:01.642 INFO:teuthology.orchestra.run.smithi006.stdout:(34/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2023-12-20T10:53:21.525 INFO:teuthology.orchestra.run.smithi006.stdout:(35/35): ceph-osd-19.0.0-282.ga8ad38c3.el9.x86_ 5.4 MB/s | 118 MB 00:21 2023-12-20T10:53:21.530 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:53:21.530 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.8 MB/s | 134 MB 00:23 2023-12-20T10:53:21.530 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:53:21.576 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:53:21.576 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:53:22.659 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:53:22.660 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:53:24.609 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:53:24.677 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-20T10:53:24.710 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-20T10:53:24.742 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-20T10:53:24.839 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-20T10:53:24.915 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-20T10:53:24.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-20T10:53:25.045 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-20T10:53:25.105 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-20T10:53:25.142 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-20T10:53:25.201 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-20T10:53:25.265 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-20T10:53:25.460 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-20T10:53:25.513 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-20T10:53:25.552 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-20T10:53:25.586 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-20T10:53:25.616 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-20T10:53:25.697 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-20T10:53:25.735 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-20T10:53:25.768 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-20T10:53:25.817 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-20T10:53:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-20T10:53:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-20T10:53:25.995 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-20T10:53:26.042 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-20T10:53:26.117 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-20T10:53:26.267 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-20T10:53:26.397 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-20T10:53:26.568 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-282.ga8ad38c3.el9.n 28/35 2023-12-20T10:53:26.596 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-282.ga8ad38c3.el9.x86_64 29/35 2023-12-20T10:53:26.623 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-282.ga8ad38c3.el9.x86_64 29/35 2023-12-20T10:53:26.623 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:53:26.623 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-20T10:53:26.624 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-20T10:53:26.624 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-20T10:53:26.624 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:53:26.715 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-20T10:53:46.226 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-20T10:53:46.257 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-282.ga8ad38c3.el9.x86_64 32/35 2023-12-20T10:53:46.285 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-282.ga8ad38c3.el9.x86_64 32/35 2023-12-20T10:53:46.285 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:53:46.285 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-20T10:53:46.286 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-20T10:53:46.286 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-20T10:53:46.286 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:53:47.216 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-282.ga8ad38c3.el9.x86_64 33/35 2023-12-20T10:53:47.239 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-282.ga8ad38c3.el9.x86_64 33/35 2023-12-20T10:53:47.240 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:53:47.240 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-20T10:53:47.240 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-20T10:53:47.240 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-20T10:53:47.240 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:53:47.450 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-282.ga8ad38c3.el9.x86_64 34/35 2023-12-20T10:53:47.474 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-282.ga8ad38c3.el9.x86_64 34/35 2023-12-20T10:53:47.474 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:53:47.474 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-20T10:53:47.474 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-20T10:53:47.474 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-20T10:53:47.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:53:47.542 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-282.ga8ad38c3.el9.x86_64 35/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:19.0.0-282.ga8ad38c3.el9.x86_64 35/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-282.ga8ad38c3.el9.x86_64 3/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-282.ga8ad38c3.el9.x86_64 4/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-282.ga8ad38c3.el9.x86_64 5/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-282.ga8ad38c3.el9.n 6/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-20T10:53:57.694 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-20T10:53:57.695 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-20T10:53:57.696 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-20T10:53:57.696 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-20T10:53:57.697 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-20T10:53:58.609 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-20T10:53:58.609 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-20T10:53:58.610 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-20T10:53:58.611 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-20T10:53:58.612 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-20T10:53:58.613 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-20T10:53:58.613 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-20T10:53:58.613 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-20T10:53:58.613 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:53:58.613 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:53:58.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2023-12-20T10:53:59.449 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:12 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:53:59.545 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:53:59.601 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:53:59.601 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:53:59.601 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:53:59.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2023-12-20T10:54:00.237 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:13 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:00.386 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:00.386 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:00.386 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:00.386 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 805 k 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 805 k 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 823 k 2023-12-20T10:54:00.387 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:00.657 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-19.0.0-282.ga8ad38c3.el9.noarch.rpm 2.9 MB/s | 805 kB 00:00 2023-12-20T10:54:00.658 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:00.658 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.9 MB/s | 805 kB 00:00 2023-12-20T10:54:00.658 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:00.664 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:00.664 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:00.670 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:00.670 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:00.749 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:01.215 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:01.305 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:01.658 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:01.984 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:01.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:01.984 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:01.984 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:01.984 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:01.984 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:02.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2023-12-20T10:54:02.778 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:15 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:02.929 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 147 k 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 147 k 2023-12-20T10:54:02.930 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 481 k 2023-12-20T10:54:02.931 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:03.498 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-19.0.0-282.ga8ad38c 260 kB/s | 147 kB 00:00 2023-12-20T10:54:03.498 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:03.498 INFO:teuthology.orchestra.run.smithi006.stdout:Total 260 kB/s | 147 kB 00:00 2023-12-20T10:54:03.499 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:03.505 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:03.505 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:03.561 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:03.562 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:03.709 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:03.772 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-282.ga8ad38c3.e 1/1 2023-12-20T10:54:03.798 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-282.ga8ad38c3.e 1/1 2023-12-20T10:54:03.798 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:54:03.798 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-20T10:54:03.798 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:04.824 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-282.ga8ad38c3.e 1/1 2023-12-20T10:54:04.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:04.824 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:04.824 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:54:04.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:04.824 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:05.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2023-12-20T10:54:05.623 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:18 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:05.720 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:05.773 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:05.774 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:05.774 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:05.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2023-12-20T10:54:06.401 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:19 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:06.552 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:06.552 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 1.8 M 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 28 k 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 15 k 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-20T10:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout:Install 7 Packages 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.1 M 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 8.7 M 2023-12-20T10:54:06.554 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:07.091 INFO:teuthology.orchestra.run.smithi006.stdout:(1/7): ceph-prometheus-alerts-19.0.0-282.ga8ad3 40 kB/s | 15 kB 00:00 2023-12-20T10:54:07.116 INFO:teuthology.orchestra.run.smithi006.stdout:(2/7): ceph-grafana-dashboards-19.0.0-282.ga8ad 70 kB/s | 28 kB 00:00 2023-12-20T10:54:07.241 INFO:teuthology.orchestra.run.smithi006.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 60 kB/s | 9.0 kB 00:00 2023-12-20T10:54:07.292 INFO:teuthology.orchestra.run.smithi006.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 233 kB/s | 41 kB 00:00 2023-12-20T10:54:07.317 INFO:teuthology.orchestra.run.smithi006.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 408 kB/s | 31 kB 00:00 2023-12-20T10:54:07.384 INFO:teuthology.orchestra.run.smithi006.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2023-12-20T10:54:07.426 INFO:teuthology.orchestra.run.smithi006.stdout:(7/7): ceph-mgr-dashboard-19.0.0-282.ga8ad38c3. 2.5 MB/s | 1.8 MB 00:00 2023-12-20T10:54:07.430 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:07.430 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-12-20T10:54:07.430 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:07.437 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:07.438 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:07.507 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:07.507 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:07.829 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:07.902 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-20T10:54:07.936 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-20T10:54:07.984 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-20T10:54:08.023 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-20T10:54:08.070 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-282.ga8ad38c3.el9.no 5/7 2023-12-20T10:54:08.319 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-282.ga8ad38c3.el9.n 6/7 2023-12-20T10:54:08.355 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-282.ga8ad38c3.el9.noarch 7/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-282.ga8ad38c3.el9.noarch 7/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-282.ga8ad38c3.el9.n 1/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-282.ga8ad38c3.el9.noarch 2/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-282.ga8ad38c3.el9.no 3/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-20T10:54:08.930 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-20T10:54:09.241 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-20T10:54:09.242 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-20T10:54:09.243 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:09.243 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:09.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-20T10:54:10.050 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:23 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:10.202 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout:=========================================================================================== 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout:=========================================================================================== 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 7.4 M 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-20T10:54:10.203 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout:=========================================================================================== 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2023-12-20T10:54:10.204 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:10.205 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 37 M 2023-12-20T10:54:10.205 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 209 M 2023-12-20T10:54:10.205 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:10.608 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 246 kB/s | 32 kB 00:00 2023-12-20T10:54:10.641 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 519 kB/s | 17 kB 00:00 2023-12-20T10:54:10.667 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2023-12-20T10:54:10.834 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2023-12-20T10:54:10.976 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-12-20T10:54:11.018 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2023-12-20T10:54:11.076 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2023-12-20T10:54:11.443 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2023-12-20T10:54:11.468 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2023-12-20T10:54:11.494 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-12-20T10:54:11.719 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 983 kB/s | 245 kB 00:00 2023-12-20T10:54:14.186 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 2.0 MB/s | 7.4 MB 00:03 2023-12-20T10:54:14.188 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:14.189 INFO:teuthology.orchestra.run.smithi006.stdout:Total 9.3 MB/s | 37 MB 00:03 2023-12-20T10:54:14.203 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:14.232 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:14.232 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:14.636 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:14.636 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:15.372 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:15.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-20T10:54:15.482 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-20T10:54:15.580 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-20T10:54:15.618 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-20T10:54:15.803 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-20T10:54:15.834 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-20T10:54:15.917 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-20T10:54:16.603 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-20T10:54:16.717 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-20T10:54:18.037 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-20T10:54:19.110 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-20T10:54:19.141 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-282.ga8ad38 12/12 2023-12-20T10:54:21.775 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-282.ga8ad38 12/12 2023-12-20T10:54:21.775 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-282.ga8ad38 1/12 2023-12-20T10:54:21.775 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-20T10:54:21.775 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-20T10:54:21.775 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-20T10:54:21.776 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-20T10:54:22.278 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-20T10:54:22.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-20T10:54:22.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-20T10:54:22.279 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-20T10:54:22.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:22.279 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:22.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2023-12-20T10:54:23.189 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:36 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:23.344 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:23.345 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2023-12-20T10:54:23.345 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:23.345 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2023-12-20T10:54:23.345 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 50 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-20T10:54:23.346 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:23.347 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2023-12-20T10:54:23.348 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2023-12-20T10:54:23.348 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:23.533 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 223 kB/s | 24 kB 00:00 2023-12-20T10:54:23.583 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 113 kB/s | 18 kB 00:00 2023-12-20T10:54:23.633 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2023-12-20T10:54:23.650 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-20T10:54:23.678 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.0 MB/s | 211 kB 00:00 2023-12-20T10:54:23.700 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 876 kB/s | 43 kB 00:00 2023-12-20T10:54:23.867 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 168 kB/s | 32 kB 00:00 2023-12-20T10:54:23.893 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2023-12-20T10:54:23.959 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): ceph-mgr-rook-19.0.0-282.ga8ad38c3.el9. 94 kB/s | 50 kB 00:00 2023-12-20T10:54:24.027 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 1.2 MB/s | 197 kB 00:00 2023-12-20T10:54:24.091 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2023-12-20T10:54:24.202 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2023-12-20T10:54:24.277 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2023-12-20T10:54:24.279 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:24.279 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2023-12-20T10:54:24.279 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:24.295 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:24.295 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:24.406 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:24.407 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:24.943 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:25.123 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-20T10:54:25.180 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-20T10:54:25.236 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-20T10:54:25.272 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-20T10:54:25.314 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-20T10:54:25.410 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-20T10:54:25.521 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-20T10:54:25.574 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-20T10:54:26.339 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-20T10:54:26.378 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-20T10:54:26.412 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-20T10:54:26.460 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-20T10:54:26.481 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-282.ga8ad38c3.el9.noarch 13/13 2023-12-20T10:54:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-282.ga8ad38c3.el9.noarch 13/13 2023-12-20T10:54:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-282.ga8ad38c3.el9.noarch 1/13 2023-12-20T10:54:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-20T10:54:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-20T10:54:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-20T10:54:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:27.684 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-20T10:54:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-20T10:54:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-20T10:54:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:27.686 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:27.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2023-12-20T10:54:28.501 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:41 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:28.656 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:28.656 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 133 k 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:28.657 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2023-12-20T10:54:28.658 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2023-12-20T10:54:28.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:28.658 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.8 M 2023-12-20T10:54:28.658 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2023-12-20T10:54:28.658 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:29.012 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.2 MB/s | 228 kB 00:00 2023-12-20T10:54:29.163 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-12-20T10:54:29.338 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): ceph-mgr-cephadm-19.0.0-282.ga8ad38c3.el 261 kB/s | 133 kB 00:00 2023-12-20T10:54:30.255 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 53 kB/s | 58 kB 00:01 2023-12-20T10:54:30.422 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 47 kB/s | 51 kB 00:01 2023-12-20T10:54:30.755 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 315 kB/s | 548 kB 00:01 2023-12-20T10:54:30.760 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:30.760 INFO:teuthology.orchestra.run.smithi006.stdout:Total 3.2 MB/s | 6.8 MB 00:02 2023-12-20T10:54:30.761 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:30.772 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:30.772 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:30.874 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:30.875 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:31.216 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:31.405 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-20T10:54:31.458 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-20T10:54:32.015 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-20T10:54:32.096 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-20T10:54:32.164 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-20T10:54:32.196 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 6/6 2023-12-20T10:54:33.004 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 6/6 2023-12-20T10:54:33.004 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 1/6 2023-12-20T10:54:33.004 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-20T10:54:33.004 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-20T10:54:33.004 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-20T10:54:33.004 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-20T10:54:33.340 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-20T10:54:33.340 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:33.340 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:33.340 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:33.340 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-20T10:54:33.341 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-20T10:54:33.341 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-20T10:54:33.341 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-20T10:54:33.341 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-20T10:54:33.341 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:33.341 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:33.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2023-12-20T10:54:34.155 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:47 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:34.306 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 858 k 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2023-12-20T10:54:34.307 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:34.308 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 937 k 2023-12-20T10:54:34.308 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 3.1 M 2023-12-20T10:54:34.308 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:34.517 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 377 kB/s | 79 kB 00:00 2023-12-20T10:54:35.201 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-fuse-19.0.0-282.ga8ad38c3.el9.x86_6 963 kB/s | 858 kB 00:00 2023-12-20T10:54:35.202 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:35.202 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.0 MB/s | 937 kB 00:00 2023-12-20T10:54:35.203 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:35.228 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:35.228 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:35.314 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:35.314 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:35.649 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-20T10:54:35.721 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/2 2023-12-20T10:54:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/2 2023-12-20T10:54:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/2 2023-12-20T10:54:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-20T10:54:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:36.743 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:36.743 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-20T10:54:36.744 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:36.744 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:36.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2023-12-20T10:54:37.579 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:50 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:37.730 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:37.731 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:37.731 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:54:37.731 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:37.731 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:37.731 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-282.ga8ad38c3.el9 ceph-noarch 262 k 2023-12-20T10:54:37.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 262 k 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.2 M 2023-12-20T10:54:37.732 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:38.522 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-volume-19.0.0-282.ga8ad38c3.el9.noarch.rpm 332 kB/s | 262 kB 00:00 2023-12-20T10:54:38.523 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:38.523 INFO:teuthology.orchestra.run.smithi006.stdout:Total 331 kB/s | 262 kB 00:00 2023-12-20T10:54:38.523 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:38.532 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:38.533 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:38.574 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:38.575 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:38.807 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:38.898 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:38.915 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:54:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-20T10:54:38.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:40.076 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-282.ga8ad38c3.el9.noarch 1/1 2023-12-20T10:54:40.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:40.076 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-282.ga8ad38c3.el9.noarch 2023-12-20T10:54:40.077 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:40.077 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:40.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2023-12-20T10:54:40.945 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:53 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:41.096 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:41.096 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 127 k 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 127 k 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 456 k 2023-12-20T10:54:41.097 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:41.410 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-19.0.0-282.ga8ad38c3.el9.x86_64. 406 kB/s | 127 kB 00:00 2023-12-20T10:54:41.410 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:41.410 INFO:teuthology.orchestra.run.smithi006.stdout:Total 404 kB/s | 127 kB 00:00 2023-12-20T10:54:41.410 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:41.417 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:41.417 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:41.476 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:41.476 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:41.627 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:41.737 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:42.536 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:42.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:42.537 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:42.537 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:54:42.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:42.537 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:42.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2023-12-20T10:54:43.359 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:56 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:43.456 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:43.510 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:43.511 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:43.511 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:43.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2023-12-20T10:54:44.155 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:57 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 31 k 2023-12-20T10:54:44.306 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 31 k 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 140 k 2023-12-20T10:54:44.307 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:44.555 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-19.0.0-282.ga8ad38c3.el9.x86_64 126 kB/s | 31 kB 00:00 2023-12-20T10:54:44.556 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:44.556 INFO:teuthology.orchestra.run.smithi006.stdout:Total 125 kB/s | 31 kB 00:00 2023-12-20T10:54:44.556 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:44.559 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:44.559 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:44.595 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:44.596 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:44.689 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:44.805 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:45.203 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:45.486 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:45.486 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:45.486 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:45.486 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:54:45.486 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:45.486 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:45.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2023-12-20T10:54:46.227 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:59 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:46.325 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:46.379 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:46.380 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:46.380 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:46.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2023-12-20T10:54:47.018 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:00 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:47.116 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:47.169 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:47.170 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:47.170 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:47.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2023-12-20T10:54:47.805 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:00 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:47.902 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:47.956 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:47.956 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:47.957 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:48.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2023-12-20T10:54:48.600 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:01 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:48.699 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:48.753 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:48.754 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:48.754 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:48.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2023-12-20T10:54:49.400 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:02 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:49.497 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:49.551 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:49.552 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:49.552 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:49.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2023-12-20T10:54:50.197 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:03 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:50.296 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 is already installed. 2023-12-20T10:54:50.351 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:50.352 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-20T10:54:50.352 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:50.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2023-12-20T10:54:51.003 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:04 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:51.155 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:51.155 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-12-20T10:54:51.155 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 86 k 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 86 k 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 256 k 2023-12-20T10:54:51.156 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:51.566 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-19.0.0-282.ga8ad38c3.el9.x86_64.rpm 210 kB/s | 86 kB 00:00 2023-12-20T10:54:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:Total 209 kB/s | 86 kB 00:00 2023-12-20T10:54:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:51.575 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:51.635 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:51.779 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:51.917 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:52.314 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:52.627 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:52.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:52.627 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:52.627 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:54:52.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:52.627 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:52.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2023-12-20T10:54:53.424 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:06 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:53.576 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 3.0 M 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:53.577 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.0 M 2023-12-20T10:54:53.578 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 14 M 2023-12-20T10:54:53.578 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:54.477 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-19.0.0-282.ga8ad38c3.el9.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2023-12-20T10:54:54.477 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:54.478 INFO:teuthology.orchestra.run.smithi006.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2023-12-20T10:54:54.478 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:54.488 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:54.488 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:54.566 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:54.566 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:55.019 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:55.086 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:55.110 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:55.111 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-20T10:54:55.111 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-20T10:54:55.111 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-20T10:54:55.111 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-20T10:54:55.111 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:56.293 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:56.293 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:56.293 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:56.293 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:54:56.294 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:56.294 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:56.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2023-12-20T10:54:57.113 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:10 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:54:57.264 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:54:57.264 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 169 k 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 169 k 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 537 k 2023-12-20T10:54:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:54:57.855 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-19.0.0-282.ga8ad38c3.el9.x86_64.rpm 287 kB/s | 169 kB 00:00 2023-12-20T10:54:57.856 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:54:57.856 INFO:teuthology.orchestra.run.smithi006.stdout:Total 286 kB/s | 169 kB 00:00 2023-12-20T10:54:57.856 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:54:57.865 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:54:57.865 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:54:57.925 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:54:57.926 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:54:58.096 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:54:58.205 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:58.650 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:58.978 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/1 2023-12-20T10:54:58.978 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:58.978 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:54:58.978 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:54:58.978 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:54:58.979 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:54:59.205 DEBUG:teuthology.parallel:result is None 2023-12-20T10:54:59.206 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=a8ad38c3a13ec7dc3249f402106905f8db892f83 2023-12-20T10:54:59.422 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-20T10:54:59.443 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-282.ga8ad38c3.el9 2023-12-20T10:54:59.444 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-282.ga8ad38c3.el9 2023-12-20T10:54:59.444 INFO:teuthology.task.install:The correct ceph version 19.0.0-282.ga8ad38c3 is installed. 2023-12-20T10:54:59.445 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-20T10:54:59.446 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:54:59.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-20T10:54:59.515 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-20T10:54:59.516 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:54:59.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2023-12-20T10:54:59.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-20T10:54:59.656 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-20T10:54:59.656 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:54:59.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-20T10:54:59.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-20T10:54:59.798 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-20T10:54:59.799 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:54:59.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2023-12-20T10:54:59.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-20T10:54:59.942 INFO:teuthology.run_tasks:Running task ceph... 2023-12-20T10:55:00.009 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-20T10:55:00.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2023-12-20T10:55:00.040 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-20T10:55:00.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-20T10:55:00.111 INFO:tasks.ceph:Creating extra log directories... 2023-12-20T10:55:00.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-20T10:55:00.191 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-20T10:55:00.191 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-20T10:55:00.191 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823', 'branch': 'wip-matanb-crimson-only-submit-error-cleanup', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': None, 'first_in_suite': False, 'job_id': '7497823', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-only-submit-error-cleanup'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83'}}, 'workunit': {'branch': 'wip-matanb-crimson-testing-misc', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 241, 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-testing-misc', 'suite_path': '/home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/Matan-B/ceph.git', 'suite_sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbLJJMp69XYhsm0XJoKroTJ3395FUcOJ4LWihlPp5H9DghW8UMjZ36ohYXosJASFQtlHNVz8rZw1l2vGy5X6Zg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a8ad38c3a13ec7dc3249f402106905f8db892f83', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-20_10:24:11', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-20T10:55:00.192 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-20T10:55:00.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2023-12-20T10:55:00.310 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:00.310 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2023-12-20T10:55:00.365 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-20T10:55:00.366 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2023-12-20T10:55:00.421 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-20T10:55:00.421 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-20T10:55:00.421 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-20T10:55:00.421 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-20T10:55:00.421 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-20T10:55:00.421 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-20 10:54:55.361605685 +0000 2023-12-20T10:55:00.422 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-20 10:50:49.330574447 +0000 2023-12-20T10:55:00.422 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-20 10:50:49.330574447 +0000 2023-12-20T10:55:00.422 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-20 10:50:49.330574447 +0000 2023-12-20T10:55:00.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-20T10:55:00.491 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-20T10:55:00.491 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-20T10:55:00.491 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000190858 s, 2.7 MB/s 2023-12-20T10:55:00.492 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-20T10:55:00.549 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-20 10:54:55.362605674 +0000 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-20 10:50:49.838568823 +0000 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-20 10:50:49.838568823 +0000 2023-12-20T10:55:00.604 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-20 10:50:49.838568823 +0000 2023-12-20T10:55:00.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-20T10:55:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-20T10:55:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-20T10:55:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000166909 s, 3.1 MB/s 2023-12-20T10:55:00.674 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-20T10:55:00.730 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2023-12-20T10:55:00.785 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-20T10:55:00.785 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-20T10:55:00.785 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-20T10:55:00.785 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-20T10:55:00.785 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-20T10:55:00.786 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-20 10:54:55.362605674 +0000 2023-12-20T10:55:00.786 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-20 10:50:50.338563288 +0000 2023-12-20T10:55:00.786 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-20 10:50:50.338563288 +0000 2023-12-20T10:55:00.786 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-20 10:50:50.338563288 +0000 2023-12-20T10:55:00.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-20T10:55:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-20T10:55:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-20T10:55:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000189787 s, 2.7 MB/s 2023-12-20T10:55:00.857 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-20T10:55:00.913 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-20 10:54:55.362605674 +0000 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-20 10:50:50.850557621 +0000 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-20 10:50:50.850557621 +0000 2023-12-20T10:55:00.969 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-20 10:50:50.850557621 +0000 2023-12-20T10:55:00.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-20T10:55:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-20T10:55:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-20T10:55:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000224624 s, 2.3 MB/s 2023-12-20T10:55:01.039 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-20T10:55:01.096 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-20T10:55:01.096 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-20T10:55:01.096 INFO:tasks.ceph:Generating config... 2023-12-20T10:55:01.097 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-20T10:55:01.097 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-20T10:55:01.097 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] err to stderr = True 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] log flush on exit = True 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] log to file = False 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] log to stderr = True 2023-12-20T10:55:01.098 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-20T10:55:01.099 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-20T10:55:01.099 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-20T10:55:01.099 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-20T10:55:01.099 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-20T10:55:01.099 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] auth client required = none 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] auth cluster required = none 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] auth service required = none 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] auth supported = none 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] debug asok = 0/0 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] debug auth = 0/0 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-12-20T10:55:01.099 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug client = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug context = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug crush = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug filer = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug journal = 0/0 2023-12-20T10:55:01.100 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mds = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug mon = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug monc = 0/0 2023-12-20T10:55:01.101 INFO:tasks.ceph:[global] debug ms = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug osd = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rados = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug refs = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug striper = 0/0 2023-12-20T10:55:01.102 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] debug timer = 0/0 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] debug tp = 0/0 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] ms client mode = crc 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-12-20T10:55:01.103 INFO:tasks.ceph:[global] ms service mode = crc 2023-12-20T10:55:01.103 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-20T10:55:01.103 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-20T10:55:01.103 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-12-20T10:55:01.103 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-12-20T10:55:01.103 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-12-20T10:55:01.103 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-12-20T10:55:01.104 INFO:tasks.ceph:Setting up mon.a... 2023-12-20T10:55:01.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-20T10:55:01.198 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2023-12-20T10:55:01.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-20T10:55:01.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-20T10:55:01.285 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6')] 2023-12-20T10:55:01.285 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-12-20T10:55:01.286 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:01.287 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-20T10:55:01.340 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:01.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:01.424+0000 7f10ad8fee80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:01.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:01.424+0000 7f10ad8fee80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:01.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:01.424+0000 7f10ad8fee80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:01.427 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:01.427 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 2b0f1823-0dfb-4950-a963-70056496e998 2023-12-20T10:55:01.427 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = pacific 2023-12-20T10:55:01.427 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 2b0f1823-0dfb-4950-a963-70056496e998 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2023-12-20T10:55:01.425998+0000 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:created 2023-12-20T10:55:01.425998+0000 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 16 (pacific) 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-12-20T10:55:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-20T10:55:01.429 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-20T10:55:01.445 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2b0f1823-0dfb-4950-a963-70056496e998... 2023-12-20T10:55:01.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-20T10:55:01.545 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = none 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2023-12-20T10:55:01.546 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2023-12-20T10:55:01.547 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.6 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: auth client required = none 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: auth cluster required = none 2023-12-20T10:55:01.548 INFO:teuthology.orchestra.run.smithi006.stdout: auth service required = none 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug asok = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug bdev = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug buffer = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug civetweb = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug compressor = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug context = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug crush = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug crypto = 0/0 2023-12-20T10:55:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: debug dpdk = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug eventtrace = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug filer = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug filestore = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug finisher = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug fuse = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug heartbeatmap = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug javaclient = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug journal = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug journaler = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug kstore = 0/0 2023-12-20T10:55:01.550 INFO:teuthology.orchestra.run.smithi006.stdout: debug leveldb = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug lockdep = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds locker = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds log = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds log expire = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds migrator = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug memdb = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgrc = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug objclass = 0/0 2023-12-20T10:55:01.551 INFO:teuthology.orchestra.run.smithi006.stdout: debug objectcacher = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug objecter = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug optracker = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug perfcounter = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rados = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd mirror = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd replay = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug refs = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rgw = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rgw sync = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug striper = 0/0 2023-12-20T10:55:01.552 INFO:teuthology.orchestra.run.smithi006.stdout: debug throttle = 0/0 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: debug timer = 0/0 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: debug tp = 0/0 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: ms client mode = crc 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: ms cluster mode = crc 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: ms mon client mode = crc 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: ms mon cluster mode = crc 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: ms mon service mode = crc 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: ms service mode = crc 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 2b0f1823-0dfb-4950-a963-70056496e998 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2023-12-20T10:55:01.553 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2023-12-20T10:55:01.554 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = snappy 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore rocksdb cf = False 2023-12-20T10:55:01.555 INFO:teuthology.orchestra.run.smithi006.stdout: crimson seastar smp = 3 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug filestore = 0/0 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug journal = 0/0 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 0/0 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 0/0 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: err to stderr = True 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: log flush on exit = True 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: log to file = False 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = True 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2023-12-20T10:55:01.556 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2023-12-20T10:55:01.557 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2023-12-20T10:55:01.557 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2023-12-20T10:55:01.557 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2023-12-20T10:55:01.557 INFO:teuthology.orchestra.run.smithi006.stdout: osd shutdown pgref assert = True 2023-12-20T10:55:01.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 0/0 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 0/0 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 0/0 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2023-12-20T10:55:01.558 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2023-12-20T10:55:01.559 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default crimson = True 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-20T10:55:01.560 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2023-12-20T10:55:01.563 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-20T10:55:01.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-20T10:55:01.628 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-20T10:55:01.629 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:01.629 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-20T10:55:01.643 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:01.644 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-20T10:55:01.699 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:55:01.699 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:01.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-20T10:55:01.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-20T10:55:01.785 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:01.785 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:01.841 INFO:tasks.ceph:Setting up mon nodes... 2023-12-20T10:55:01.841 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-20T10:55:01.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-20T10:55:01.948 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-20T10:55:01.951 INFO:tasks.ceph:Setting up mds nodes... 2023-12-20T10:55:01.951 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-20T10:55:01.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-20T10:55:02.003 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-20T10:55:02.022 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-20T10:55:02.022 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-20T10:55:02.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-20T10:55:02.052 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-20T10:55:02.052 INFO:tasks.ceph:role: osd.0 2023-12-20T10:55:02.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:55:02.053 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-20T10:55:02.130 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-20T10:55:02.131 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-20T10:55:02.131 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-20T10:55:02.131 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-20T10:55:02.131 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-20T10:55:02.131 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-12-20T10:55:02.131 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-20T10:55:02.132 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-20T10:55:02.132 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-20T10:55:02.132 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-20T10:55:02.650 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-12-20T10:55:02.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-12-20T10:55:02.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-20T10:55:02.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-20T10:55:02.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-20T10:55:02.905 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-20T10:55:02.905 INFO:tasks.ceph:role: osd.1 2023-12-20T10:55:02.905 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:55:02.905 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-20T10:55:02.984 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-20T10:55:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-20T10:55:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-20T10:55:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-20T10:55:02.985 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-20T10:55:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-12-20T10:55:02.985 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-20T10:55:02.985 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-20T10:55:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-20T10:55:02.985 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-20T10:55:03.434 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-12-20T10:55:03.436 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-12-20T10:55:03.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-20T10:55:03.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-20T10:55:03.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-20T10:55:03.652 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-20T10:55:03.652 INFO:tasks.ceph:role: osd.2 2023-12-20T10:55:03.652 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:55:03.652 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-20T10:55:03.733 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-20T10:55:04.212 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-12-20T10:55:04.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-12-20T10:55:04.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-20T10:55:04.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-20T10:55:04.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:04.980 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-20T10:55:04.980 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:04,979 seastar - Reactor backend: linux-aio 2023-12-20T10:55:05.000 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:04,999 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-20T10:55:05.000 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:04,999 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-20T10:55:05.001 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:04,999 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-20T10:55:05.001 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:04,999 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-20T10:55:05.002 INFO:teuthology.orchestra.run.smithi006.stderr:==80410==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-20T10:55:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:==80410==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3f4b3000; bottom 0x7f808073a000; size: 0x007dbed79000 (540072710144) 2023-12-20T10:55:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-20T10:55:05.009 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-20T10:55:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-20T10:55:05.565 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-20T10:55:05.565 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-20T10:55:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:05.565 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-20T10:55:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:05,620 seastar - Reactor backend: linux-aio 2023-12-20T10:55:05.649 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:05,647 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-20T10:55:05.649 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:05,647 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-20T10:55:05.649 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:05,647 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-20T10:55:05.656 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:05,655 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-20T10:55:05.659 INFO:teuthology.orchestra.run.smithi006.stderr:==80408==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-20T10:55:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:==80408==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3f4b3000; bottom 0x7f807e533000; size: 0x007dc0f80000 (540108390400) 2023-12-20T10:55:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-20T10:55:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-20T10:55:05.691 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0x22f0da93 0x2302905b 0x2302e48c 0x24f17d0b 0x24f1e9f6 0x24f21344 0x24f2251d 0x22a6c4db 0x22abd138 0x22ae0f65 0x2cbd3dbd 0x2cc71bca 0x2ce9614b 0x2ce97f95 0x2c8ecbb0 0x2c8eeab9 0x1ef4ecbc 0x3feaf 0x3ff5f 0x1e9f6a44 2023-12-20T10:55:05.691 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-20T10:55:05.752 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2679a282 0x2679c8cb 0x2679e6dc 0x267a0f41 0x265d191d 0x1f00b520 0x1ef7cf31 0x1ef80d72 0x2cfb5dbf 2023-12-20T10:55:05.752 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8a9508a3 0xffffffff8a9508ea 0xffffffff8b46646c 0xffffffff8b6000ea 2023-12-20T10:55:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-20T10:55:09.699 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-20T10:55:09.699 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-20T10:55:09.699 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:09.699 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:09.699 INFO:teuthology.orchestra.run.smithi006.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 2b0f1823-0dfb-4950-a963-70056496e998 2023-12-20T10:55:09.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-20T10:55:09.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:10.320 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-20T10:55:10.320 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,319 seastar - Reactor backend: linux-aio 2023-12-20T10:55:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,339 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-20T10:55:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,339 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-20T10:55:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,339 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-20T10:55:10.341 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,339 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-20T10:55:10.343 INFO:teuthology.orchestra.run.smithi006.stderr:==80592==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-20T10:55:10.349 INFO:teuthology.orchestra.run.smithi006.stderr:==80592==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdc7d3c000; bottom 0x7f9ce35ca000; size: 0x0060e4772000 (416149872640) 2023-12-20T10:55:10.349 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-20T10:55:10.349 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-20T10:55:10.375 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0x231cbe8d 0x231d0ca8 0x231d11f6 0x231db88b 0x231dcf23 0x2322c354 0x231f0705 0x231f3d2e 0x23207d1f 0x232279b1 0x23228239 0x231d5e19 0x231e5439 0x231e6b35 0x231e750e 0x231e85f9 0x232ca538 0x2312f1a9 0x231f9484 0x231faecf 0x231fbc38 0x231c3aa4 0x231c30fc 0x22ecf854 0x24f17240 0x24f1e9f6 0x24f21344 0x24f2251d 0x22a6c4db 0x22abd138 0x22ae0f65 0x2cbd3dbd 0x2cc71bca 0x2ce9614b 0x2ce97f95 0x2c8ecbb0 0x2c8eeab9 0x1f0d6240 0x1f0defdd 0x1ef4d406 0x3feaf 0x3ff5f 0x1e9f6a44 2023-12-20T10:55:10.375 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-20T10:55:10.900 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:10.900 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-20T10:55:10.900 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-20T10:55:10.900 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-20T10:55:10.901 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:10.901 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:10.962 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-20T10:55:10.963 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,962 seastar - Reactor backend: linux-aio 2023-12-20T10:55:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,980 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-20T10:55:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,980 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-20T10:55:10.981 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,980 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-20T10:55:10.988 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:10,988 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-20T10:55:10.991 INFO:teuthology.orchestra.run.smithi006.stderr:==80590==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-20T10:55:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:==80590==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdc7d3c000; bottom 0x7f9ce13c3000; size: 0x0060e6979000 (416185552896) 2023-12-20T10:55:10.997 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-20T10:55:10.998 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-20T10:55:11.025 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0x1ee87eec 0x22f963b8 0x23023cb5 0x23028e1b 0x2302e48c 0x24f17d0b 0x24f1e9f6 0x24f21344 0x24f2251d 0x22a6c4db 0x22abd138 0x22ae0f65 0x2cbd3dbd 0x2cc71bca 0x2ce9614b 0x2ce97f95 0x2c8ecbb0 0x2c8eeab9 0x1ef4ecbc 0x3feaf 0x3ff5f 0x1e9f6a44 2023-12-20T10:55:11.025 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-20T10:55:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2679a282 0x2679c8cb 0x2679e6dc 0x267a0f41 0x265d191d 0x1f00b520 0x1ef7cf31 0x1ef80d72 0x2cfb5dbf 2023-12-20T10:55:11.084 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-20T10:55:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-20T10:55:15.058 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-20T10:55:15.058 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-20T10:55:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:15.059 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:15.059 INFO:teuthology.orchestra.run.smithi006.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 2b0f1823-0dfb-4950-a963-70056496e998 2023-12-20T10:55:15.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-20T10:55:15.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:15.681 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-20T10:55:15.682 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:15,681 seastar - Reactor backend: linux-aio 2023-12-20T10:55:15.698 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:15,697 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-20T10:55:15.698 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:15,697 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-20T10:55:15.699 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:15,697 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-20T10:55:15.699 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:15,697 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-20T10:55:15.700 INFO:teuthology.orchestra.run.smithi006.stderr:==80774==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-20T10:55:15.707 INFO:teuthology.orchestra.run.smithi006.stderr:==80774==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff032b0000; bottom 0x7f91b1f99000; size: 0x006d51317000 (469513629696) 2023-12-20T10:55:15.707 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-20T10:55:15.707 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-20T10:55:15.733 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0x231e8d2c 0x231ea059 0x231eb176 0x231c96a9 0x231c9d60 0x231d7c0c 0x231d96f0 0x231dcf23 0x2322c354 0x231f0705 0x231f3d2e 0x23207d1f 0x232279b1 0x23228239 0x231d5e19 0x231e5439 0x231e6b35 0x231e750e 0x231e85f9 0x232ca538 0x2312f1a9 0x231f9484 0x231faecf 0x231fbc38 0x231c3aa4 0x231c30fc 0x22ecf854 0x24f17240 0x24f1e9f6 0x24f21344 0x24f2251d 0x22a6c4db 0x22abd138 0x22ae0f65 0x2cbd3dbd 0x2cc71bca 0x2ce9614b 0x2ce97f95 0x2c8ecbb0 0x2c8eeab9 0x1f0d6240 0x1f0defdd 0x1ef4d406 0x3feaf 0x3ff5f 0x1e9f6a44 2023-12-20T10:55:15.733 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-20T10:55:16.254 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:16.254 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-20T10:55:16.254 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-20T10:55:16.254 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-20T10:55:16.254 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:16.254 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-20T10:55:16.315 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:16,314 seastar - Reactor backend: linux-aio 2023-12-20T10:55:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:16,332 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-20T10:55:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:16,332 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-20T10:55:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:16,332 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-20T10:55:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-20 10:55:16,339 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-20T10:55:16.343 INFO:teuthology.orchestra.run.smithi006.stderr:==80772==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-20T10:55:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:==80772==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff032b0000; bottom 0x7f91afd92000; size: 0x006d5351e000 (469549309952) 2023-12-20T10:55:16.349 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-20T10:55:16.350 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-20T10:55:16.375 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cc31aa2 0x2cc32e50 0x2cc3439e 0x2cc34722 0x2cc3486c 0x2cc34cc2 0x54daf 0x22f21095 0x22f76657 0x22f96022 0x23023cb5 0x23028e1b 0x2302e48c 0x24f17d0b 0x24f1e9f6 0x24f21344 0x24f2251d 0x22a6c4db 0x22abd138 0x22ae0f65 0x2cbd3dbd 0x2cc71bca 0x2ce9614b 0x2ce97f95 0x2c8ecbb0 0x2c8eeab9 0x1ef4ecbc 0x3feaf 0x3ff5f 0x1e9f6a44 2023-12-20T10:55:16.375 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-20T10:55:20.395 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:20.395 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-20T10:55:20.395 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-20T10:55:20.395 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-20T10:55:20.396 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-20T10:55:20.396 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:20.396 INFO:teuthology.orchestra.run.smithi006.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 2b0f1823-0dfb-4950-a963-70056496e998 2023-12-20T10:55:20.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-20T10:55:20.465 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-20T10:55:20.466 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:20.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-20T10:55:20.536 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:20.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-20T10:55:20.606 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:20.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-20T10:55:20.677 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:20.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-20T10:55:20.747 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:20.747 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-20T10:55:20.803 INFO:tasks.ceph:Adding keys to all mons... 2023-12-20T10:55:20.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-20T10:55:20.872 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAFyIJlcOF6OBAAINJNRHOtVvZrRDAmpvgFtQ== 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAJyIJlLfKbKhAAgHfgJYKThvSUyOukMf+IEA== 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAPyIJlV8O+AhAAsyMdTqPsU7wPvgF9zq8QWQ== 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAUyIJl5RO9FxAAlWZazF5HmBWdDQgV3HmblQ== 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2023-12-20T10:55:20.873 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAGyIJl4qQgABAAhUXLP1hRviZ/BsezblTmzA== 2023-12-20T10:55:20.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-20T10:55:20.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-20T10:55:21.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-20T10:55:21.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-20T10:55:21.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-20T10:55:21.226 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-20T10:55:21.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-20T10:55:21.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-20T10:55:21.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:21.361+0000 7faeca84c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:21.361+0000 7faeca84c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-20T10:55:21.410 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-20T10:55:21.465 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-20T10:55:21.465 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-20T10:55:21.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-20T10:55:21.507 INFO:tasks.ceph.mon.a:Started 2023-12-20T10:55:21.507 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-20T10:55:21.508 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-20T10:55:21.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-20T10:55:21.510 INFO:tasks.ceph.mgr.x:Started 2023-12-20T10:55:21.511 INFO:tasks.ceph:Setting crush tunables to default 2023-12-20T10:55:21.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2023-12-20T10:55:21.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-20T10:55:21.591+0000 7eff93651980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.593 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-20T10:55:21.591+0000 7eff93651980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.594 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-20T10:55:21.591+0000 7eff93651980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.595 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:21.593+0000 7f0b22369f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:21.618+0000 7f173e19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:21.662+0000 7f173e19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.734 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2023-12-20T10:55:21.745 INFO:tasks.ceph:check_enable_crimson: True 2023-12-20T10:55:21.745 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-12-20T10:55:21.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-12-20T10:55:21.799 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:21.797+0000 7f0b22369f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.799 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:21.797+0000 7f0b22369f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:21.851+0000 7f06a22de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:21.858+0000 7f06a22de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:21.941 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-20T10:55:21.939+0000 7eff8a9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-12-20T10:55:21.941 INFO:tasks.ceph.mon.a.smithi006.stderr:Please be aware that this feature is experimental, untested, 2023-12-20T10:55:21.941 INFO:tasks.ceph.mon.a.smithi006.stderr:unsupported, and may result in data corruption, data loss, 2023-12-20T10:55:21.941 INFO:tasks.ceph.mon.a.smithi006.stderr:and/or irreparable damage to your cluster. Do not use 2023-12-20T10:55:21.941 INFO:tasks.ceph.mon.a.smithi006.stderr:feature with important data. 2023-12-20T10:55:21.941 INFO:tasks.ceph.mon.a.smithi006.stderr: 2023-12-20T10:55:22.272 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:22.270+0000 7f0b22369f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-20T10:55:22.326 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:22.324+0000 7f0b22369f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-20T10:55:22.747 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-20T10:55:22.747 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:22.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-20T10:55:22.779 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:22.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-20T10:55:22.850 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:22.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-20T10:55:22.907 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:22.905+0000 7f0b22369f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-20T10:55:22.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd new 7718323d-a168-42f3-9306-71d30ab24aff 0 2023-12-20T10:55:22.963 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:22.961+0000 7f0b22369f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-20T10:55:23.013 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.011+0000 7f0b22369f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-20T10:55:23.071 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.069+0000 7f0b22369f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-20T10:55:23.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:23.068+0000 7ff88eb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:23.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:23.075+0000 7ff88eb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:23.117 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.115+0000 7f0b22369f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-20T10:55:23.157 INFO:teuthology.orchestra.run.smithi006.stdout:0 2023-12-20T10:55:23.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd new 976b2df3-313d-409e-be1e-f0662daeb902 1 2023-12-20T10:55:23.226 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.225+0000 7f0b22369f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-20T10:55:23.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:23.272+0000 7fb207e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:23.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:23.279+0000 7fb207e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:23.303 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.301+0000 7f0b22369f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-20T10:55:23.351 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.349+0000 7f0b22369f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-20T10:55:23.359 INFO:teuthology.orchestra.run.smithi006.stdout:1 2023-12-20T10:55:23.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd new 9cdb10a0-9641-4e84-89a0-f29121d6b505 2 2023-12-20T10:55:23.410 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.408+0000 7f0b22369f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-20T10:55:23.461 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.459+0000 7f0b22369f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-20T10:55:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:23.474+0000 7f182f15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:23.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:23.481+0000 7f182f15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:23.561 INFO:teuthology.orchestra.run.smithi006.stdout:2 2023-12-20T10:55:23.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-12-20T10:55:23.649 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-20T10:55:23.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-12-20T10:55:23.691 INFO:tasks.ceph.osd.0:Started 2023-12-20T10:55:23.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-12-20T10:55:23.727 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-20T10:55:23.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-12-20T10:55:23.769 INFO:tasks.ceph.osd.1:Started 2023-12-20T10:55:23.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-12-20T10:55:23.809 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-20T10:55:23.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-12-20T10:55:23.852 INFO:tasks.ceph.osd.2:Started 2023-12-20T10:55:23.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-20T10:55:23.900 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:23.898+0000 7f0b22369f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-20T10:55:24.068 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.066+0000 7f0b22369f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-20T10:55:24.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:24.067+0000 7f439386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:24.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:24.078+0000 7f439386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:24.216 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.213+0000 7f0b22369f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-20T10:55:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:24.219 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"fsid":"2b0f1823-0dfb-4950-a963-70056496e998","created":"2023-12-20T10:55:21.646536+0000","modified":"2023-12-20T10:55:23.559909+0000","last_up_change":"0.000000","last_in_change":"2023-12-20T10:55:23.559909+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7718323d-a168-42f3-9306-71d30ab24aff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"976b2df3-313d-409e-be1e-f0662daeb902","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cdb10a0-9641-4e84-89a0-f29121d6b505","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-20T10:55:24.242 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-20T10:55:24.242 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-20T10:55:24.280 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.278+0000 7f0b22369f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-20T10:55:24.391 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.389+0000 7f0b22369f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-20T10:55:24.493 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.491+0000 7f0b22369f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-20T10:55:24.544 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-20T10:55:24.643 INFO:teuthology.misc.health.smithi006.stderr:2023-12-20T10:55:24.638+0000 7f950f813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:24.645 INFO:teuthology.misc.health.smithi006.stderr:2023-12-20T10:55:24.643+0000 7f950f813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:24.647 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.644+0000 7f0b22369f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-20T10:55:24.695 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.693+0000 7f0b22369f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-20T10:55:24.713 INFO:teuthology.misc.health.smithi006.stdout: 2023-12-20T10:55:24.713 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":6,"fsid":"2b0f1823-0dfb-4950-a963-70056496e998","created":"2023-12-20T10:55:21.646536+0000","modified":"2023-12-20T10:55:23.559909+0000","last_up_change":"0.000000","last_in_change":"2023-12-20T10:55:23.559909+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7718323d-a168-42f3-9306-71d30ab24aff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"976b2df3-313d-409e-be1e-f0662daeb902","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9cdb10a0-9641-4e84-89a0-f29121d6b505","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-20T10:55:24.724 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-20T10:55:24.853 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:24.851+0000 7f0b22369f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-20T10:55:25.738 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:25.736+0000 7f0b22369f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-20T10:55:26.029 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:26.026+0000 7f0b22369f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-20T10:55:26.090 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:26.088+0000 7f0b22369f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-20T10:55:26.202 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:26.200+0000 7f0b22369f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-20T10:55:27.222 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-20T10:55:27.220+0000 7f0ac1ed7640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-20T10:55:31.027 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-20T10:55:31.137 INFO:teuthology.misc.health.smithi006.stderr:2023-12-20T10:55:31.132+0000 7f8419b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:31.139 INFO:teuthology.misc.health.smithi006.stderr:2023-12-20T10:55:31.137+0000 7f8419b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:31.327 INFO:teuthology.misc.health.smithi006.stdout: 2023-12-20T10:55:31.327 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":9,"fsid":"2b0f1823-0dfb-4950-a963-70056496e998","created":"2023-12-20T10:55:21.646536+0000","modified":"2023-12-20T10:55:31.227209+0000","last_up_change":"2023-12-20T10:55:29.221187+0000","last_in_change":"2023-12-20T10:55:23.559909+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-20T10:55:30.272707+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7718323d-a168-42f3-9306-71d30ab24aff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":424353064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":424353064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":424353064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":424353064}]},"public_addr":"172.21.15.6:6802/424353064","cluster_addr":"172.21.15.6:6803/424353064","heartbeat_back_addr":"172.21.15.6:6810/424353064","heartbeat_front_addr":"172.21.15.6:6811/424353064","state":["exists","up"]},{"osd":1,"uuid":"976b2df3-313d-409e-be1e-f0662daeb902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":2746672455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2746672455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2746672455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2746672455}]},"public_addr":"172.21.15.6:6807/2746672455","cluster_addr":"172.21.15.6:6806/2746672455","heartbeat_back_addr":"172.21.15.6:6812/2746672455","heartbeat_front_addr":"172.21.15.6:6813/2746672455","state":["exists","up"]},{"osd":2,"uuid":"9cdb10a0-9641-4e84-89a0-f29121d6b505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":3314964054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3314964054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3314964054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":3314964054}]},"public_addr":"172.21.15.6:6805/3314964054","cluster_addr":"172.21.15.6:6804/3314964054","heartbeat_back_addr":"172.21.15.6:6808/3314964054","heartbeat_front_addr":"172.21.15.6:6809/3314964054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-20T10:55:31.338 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-20T10:55:31.339 INFO:tasks.ceph:Creating RBD pool 2023-12-20T10:55:31.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-20T10:55:31.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:31.442+0000 7fc1ccb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:31.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:31.450+0000 7fc1ccb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:32.237 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2023-12-20T10:55:32.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-20T10:55:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:32.377+0000 7f469139e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:32.388+0000 7f469139e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:33.247 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-20T10:55:33.265 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-20T10:55:33.266 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-20T10:55:33.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-20T10:55:33.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-20T10:55:33.266 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-20T10:55:33.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:33.402+0000 7f4a97c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:33.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:33.407+0000 7f4a97c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:33.649 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:33.661 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":4210727403},{"type":"v1","addr":"172.21.15.6:6801","nonce":4210727403}]},"active_addr":"172.21.15.6:6801/4210727403","active_change":"2023-12-20T10:55:26.213696+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2285558681}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3562557355}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":417914779}]}]} 2023-12-20T10:55:33.664 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-20T10:55:33.664 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-20T10:55:33.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-20T10:55:33.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:33.772+0000 7f94d971e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:33.777+0000 7f94d971e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:33.966 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":11,"fsid":"2b0f1823-0dfb-4950-a963-70056496e998","created":"2023-12-20T10:55:21.646536+0000","modified":"2023-12-20T10:55:33.245477+0000","last_up_change":"2023-12-20T10:55:29.221187+0000","last_in_change":"2023-12-20T10:55:23.559909+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-20T10:55:30.272707+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-20T10:55:31.664229+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7718323d-a168-42f3-9306-71d30ab24aff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":424353064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":424353064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":424353064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":424353064}]},"public_addr":"172.21.15.6:6802/424353064","cluster_addr":"172.21.15.6:6803/424353064","heartbeat_back_addr":"172.21.15.6:6810/424353064","heartbeat_front_addr":"172.21.15.6:6811/424353064","state":["exists","up"]},{"osd":1,"uuid":"976b2df3-313d-409e-be1e-f0662daeb902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":2746672455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2746672455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2746672455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2746672455}]},"public_addr":"172.21.15.6:6807/2746672455","cluster_addr":"172.21.15.6:6806/2746672455","heartbeat_back_addr":"172.21.15.6:6812/2746672455","heartbeat_front_addr":"172.21.15.6:6813/2746672455","state":["exists","up"]},{"osd":2,"uuid":"9cdb10a0-9641-4e84-89a0-f29121d6b505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":3314964054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3314964054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3314964054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":3314964054}]},"public_addr":"172.21.15.6:6805/3314964054","cluster_addr":"172.21.15.6:6804/3314964054","heartbeat_back_addr":"172.21.15.6:6808/3314964054","heartbeat_front_addr":"172.21.15.6:6809/3314964054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-20T10:55:33.978 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-20T10:55:33.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-20T10:55:34.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.085+0000 7f8f766ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.090+0000 7f8f766ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.277 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:34.277 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":12,"fsid":"2b0f1823-0dfb-4950-a963-70056496e998","created":"2023-12-20T10:55:21.646536+0000","modified":"2023-12-20T10:55:34.249874+0000","last_up_change":"2023-12-20T10:55:29.221187+0000","last_in_change":"2023-12-20T10:55:23.559909+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-20T10:55:30.272707+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-20T10:55:31.664229+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7718323d-a168-42f3-9306-71d30ab24aff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":424353064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":424353064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":424353064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":424353064}]},"public_addr":"172.21.15.6:6802/424353064","cluster_addr":"172.21.15.6:6803/424353064","heartbeat_back_addr":"172.21.15.6:6810/424353064","heartbeat_front_addr":"172.21.15.6:6811/424353064","state":["exists","up"]},{"osd":1,"uuid":"976b2df3-313d-409e-be1e-f0662daeb902","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":2746672455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2746672455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2746672455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2746672455}]},"public_addr":"172.21.15.6:6807/2746672455","cluster_addr":"172.21.15.6:6806/2746672455","heartbeat_back_addr":"172.21.15.6:6812/2746672455","heartbeat_front_addr":"172.21.15.6:6813/2746672455","state":["exists","up"]},{"osd":2,"uuid":"9cdb10a0-9641-4e84-89a0-f29121d6b505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":3314964054}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3314964054}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":3314964054}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":3314964054}]},"public_addr":"172.21.15.6:6805/3314964054","cluster_addr":"172.21.15.6:6804/3314964054","heartbeat_back_addr":"172.21.15.6:6808/3314964054","heartbeat_front_addr":"172.21.15.6:6809/3314964054","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-20T10:55:34.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-20T10:55:34.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-20T10:55:34.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-20T10:55:34.413 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.407+0000 7f4b1a8eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.407+0000 7fe765f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.407+0000 7f08e9860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.414+0000 7f4b1a8eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.414+0000 7fe765f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.414+0000 7f08e9860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.441 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-20T10:55:34.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-20T10:55:34.442 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-20T10:55:34.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-20T10:55:34.445 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-20T10:55:34.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-20T10:55:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.553+0000 7f4b6439e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.559+0000 7f4b6439e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.600+0000 7f8145b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.606+0000 7f8145b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.611+0000 7f9c6f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:34.618+0000 7f9c6f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:34.769 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-20T10:55:34.782 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-12-20T10:55:34.799 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-20T10:55:34.805 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-20T10:55:34.813 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-12-20T10:55:34.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-12-20T10:55:34.818 DEBUG:teuthology.parallel:result is None 2023-12-20T10:55:35.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-20T10:55:35.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-20T10:55:35.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:35.905+0000 7f280d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:35.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:35.913+0000 7f280d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:35.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:35.946+0000 7fe5cf51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:35.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:35.952+0000 7fe5cf51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:36.122 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-20T10:55:36.134 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-12-20T10:55:36.159 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-20T10:55:36.171 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-12-20T10:55:37.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-20T10:55:37.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-20T10:55:37.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.259+0000 7fd3d379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.267+0000 7fd3d379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.296+0000 7f233d017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.302+0000 7f233d017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.458 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-20T10:55:37.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-12-20T10:55:37.470 DEBUG:teuthology.parallel:result is None 2023-12-20T10:55:37.491 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-20T10:55:37.503 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-12-20T10:55:37.503 DEBUG:teuthology.parallel:result is None 2023-12-20T10:55:37.503 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-20T10:55:37.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-20T10:55:37.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.651+0000 7f76cb908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.656+0000 7f76cb908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.837 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-12-20T10:55:37.849 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-20T10:55:36.217904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80768,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034752,"statfs":{"total":289910292480,"available":289827586048,"internally_reserved":0,"allocated":278528,"data_stored":1229413,"data_compressed":59928,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.945294"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.269931+0000","last_change":"2023-12-20T10:55:33.349913+0000","last_active":"2023-12-20T10:55:34.269931+0000","last_peered":"2023-12-20T10:55:34.269931+0000","last_clean":"2023-12-20T10:55:34.269931+0000","last_became_active":"2023-12-20T10:55:33.347545+0000","last_became_peered":"2023-12-20T10:55:33.347545+0000","last_unstale":"2023-12-20T10:55:34.269931+0000","last_undegraded":"2023-12-20T10:55:34.269931+0000","last_fullsized":"2023-12-20T10:55:34.269931+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.268311+0000","last_change":"2023-12-20T10:55:33.345970+0000","last_active":"2023-12-20T10:55:34.268311+0000","last_peered":"2023-12-20T10:55:34.268311+0000","last_clean":"2023-12-20T10:55:34.268311+0000","last_became_active":"2023-12-20T10:55:33.342990+0000","last_became_peered":"2023-12-20T10:55:33.342990+0000","last_unstale":"2023-12-20T10:55:34.268311+0000","last_undegraded":"2023-12-20T10:55:34.268311+0000","last_fullsized":"2023-12-20T10:55:34.268311+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.271908+0000","last_change":"2023-12-20T10:55:33.330650+0000","last_active":"2023-12-20T10:55:34.271908+0000","last_peered":"2023-12-20T10:55:34.271908+0000","last_clean":"2023-12-20T10:55:34.271908+0000","last_became_active":"2023-12-20T10:55:33.325663+0000","last_became_peered":"2023-12-20T10:55:33.325663+0000","last_unstale":"2023-12-20T10:55:34.271908+0000","last_undegraded":"2023-12-20T10:55:34.271908+0000","last_fullsized":"2023-12-20T10:55:34.271908+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.268788+0000","last_change":"2023-12-20T10:55:33.373231+0000","last_active":"2023-12-20T10:55:34.268788+0000","last_peered":"2023-12-20T10:55:34.268788+0000","last_clean":"2023-12-20T10:55:34.268788+0000","last_became_active":"2023-12-20T10:55:33.369195+0000","last_became_peered":"2023-12-20T10:55:33.369195+0000","last_unstale":"2023-12-20T10:55:34.268788+0000","last_undegraded":"2023-12-20T10:55:34.268788+0000","last_fullsized":"2023-12-20T10:55:34.268788+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-20T10:55:33.357504+0000","last_change":"2023-12-20T10:55:33.357561+0000","last_active":"2023-12-20T10:55:33.357504+0000","last_peered":"2023-12-20T10:55:33.357504+0000","last_clean":"2023-12-20T10:55:32.234916+0000","last_became_active":"2023-12-20T10:55:33.355764+0000","last_became_peered":"2023-12-20T10:55:33.355764+0000","last_unstale":"2023-12-20T10:55:33.357504+0000","last_undegraded":"2023-12-20T10:55:33.357504+0000","last_fullsized":"2023-12-20T10:55:33.357504+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.270671+0000","last_change":"2023-12-20T10:55:33.366776+0000","last_active":"2023-12-20T10:55:34.270671+0000","last_peered":"2023-12-20T10:55:34.270671+0000","last_clean":"2023-12-20T10:55:34.270671+0000","last_became_active":"2023-12-20T10:55:33.364836+0000","last_became_peered":"2023-12-20T10:55:33.364836+0000","last_unstale":"2023-12-20T10:55:34.270671+0000","last_undegraded":"2023-12-20T10:55:34.270671+0000","last_fullsized":"2023-12-20T10:55:34.270671+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.270562+0000","last_change":"2023-12-20T10:55:33.320937+0000","last_active":"2023-12-20T10:55:34.270562+0000","last_peered":"2023-12-20T10:55:34.270562+0000","last_clean":"2023-12-20T10:55:34.270562+0000","last_became_active":"2023-12-20T10:55:33.318837+0000","last_became_peered":"2023-12-20T10:55:33.318837+0000","last_unstale":"2023-12-20T10:55:34.270562+0000","last_undegraded":"2023-12-20T10:55:34.270562+0000","last_fullsized":"2023-12-20T10:55:34.270562+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.272225+0000","last_change":"2023-12-20T10:55:33.310138+0000","last_active":"2023-12-20T10:55:34.272225+0000","last_peered":"2023-12-20T10:55:34.272225+0000","last_clean":"2023-12-20T10:55:34.272225+0000","last_became_active":"2023-12-20T10:55:33.306467+0000","last_became_peered":"2023-12-20T10:55:33.306467+0000","last_unstale":"2023-12-20T10:55:34.272225+0000","last_undegraded":"2023-12-20T10:55:34.272225+0000","last_fullsized":"2023-12-20T10:55:34.272225+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.269373+0000","last_change":"2023-12-20T10:55:32.299593+0000","last_active":"2023-12-20T10:55:34.269373+0000","last_peered":"2023-12-20T10:55:34.269373+0000","last_clean":"2023-12-20T10:55:34.269373+0000","last_became_active":"2023-12-20T10:55:32.293211+0000","last_became_peered":"2023-12-20T10:55:32.293211+0000","last_unstale":"2023-12-20T10:55:34.269373+0000","last_undegraded":"2023-12-20T10:55:34.269373+0000","last_fullsized":"2023-12-20T10:55:34.269373+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:31.227209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:31.227209+0000","last_clean_scrub_stamp":"2023-12-20T10:55:31.227209+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607364,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-20T10:55:37.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-20T10:55:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.957+0000 7f1b00f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:37.962+0000 7f1b00f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:38.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-12-20T10:55:38.156 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-20T10:55:36.217904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80768,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034752,"statfs":{"total":289910292480,"available":289827586048,"internally_reserved":0,"allocated":278528,"data_stored":1229413,"data_compressed":59928,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.945294"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.269931+0000","last_change":"2023-12-20T10:55:33.349913+0000","last_active":"2023-12-20T10:55:34.269931+0000","last_peered":"2023-12-20T10:55:34.269931+0000","last_clean":"2023-12-20T10:55:34.269931+0000","last_became_active":"2023-12-20T10:55:33.347545+0000","last_became_peered":"2023-12-20T10:55:33.347545+0000","last_unstale":"2023-12-20T10:55:34.269931+0000","last_undegraded":"2023-12-20T10:55:34.269931+0000","last_fullsized":"2023-12-20T10:55:34.269931+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.268311+0000","last_change":"2023-12-20T10:55:33.345970+0000","last_active":"2023-12-20T10:55:34.268311+0000","last_peered":"2023-12-20T10:55:34.268311+0000","last_clean":"2023-12-20T10:55:34.268311+0000","last_became_active":"2023-12-20T10:55:33.342990+0000","last_became_peered":"2023-12-20T10:55:33.342990+0000","last_unstale":"2023-12-20T10:55:34.268311+0000","last_undegraded":"2023-12-20T10:55:34.268311+0000","last_fullsized":"2023-12-20T10:55:34.268311+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.271908+0000","last_change":"2023-12-20T10:55:33.330650+0000","last_active":"2023-12-20T10:55:34.271908+0000","last_peered":"2023-12-20T10:55:34.271908+0000","last_clean":"2023-12-20T10:55:34.271908+0000","last_became_active":"2023-12-20T10:55:33.325663+0000","last_became_peered":"2023-12-20T10:55:33.325663+0000","last_unstale":"2023-12-20T10:55:34.271908+0000","last_undegraded":"2023-12-20T10:55:34.271908+0000","last_fullsized":"2023-12-20T10:55:34.271908+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.268788+0000","last_change":"2023-12-20T10:55:33.373231+0000","last_active":"2023-12-20T10:55:34.268788+0000","last_peered":"2023-12-20T10:55:34.268788+0000","last_clean":"2023-12-20T10:55:34.268788+0000","last_became_active":"2023-12-20T10:55:33.369195+0000","last_became_peered":"2023-12-20T10:55:33.369195+0000","last_unstale":"2023-12-20T10:55:34.268788+0000","last_undegraded":"2023-12-20T10:55:34.268788+0000","last_fullsized":"2023-12-20T10:55:34.268788+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-20T10:55:33.357504+0000","last_change":"2023-12-20T10:55:33.357561+0000","last_active":"2023-12-20T10:55:33.357504+0000","last_peered":"2023-12-20T10:55:33.357504+0000","last_clean":"2023-12-20T10:55:32.234916+0000","last_became_active":"2023-12-20T10:55:33.355764+0000","last_became_peered":"2023-12-20T10:55:33.355764+0000","last_unstale":"2023-12-20T10:55:33.357504+0000","last_undegraded":"2023-12-20T10:55:33.357504+0000","last_fullsized":"2023-12-20T10:55:33.357504+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.270671+0000","last_change":"2023-12-20T10:55:33.366776+0000","last_active":"2023-12-20T10:55:34.270671+0000","last_peered":"2023-12-20T10:55:34.270671+0000","last_clean":"2023-12-20T10:55:34.270671+0000","last_became_active":"2023-12-20T10:55:33.364836+0000","last_became_peered":"2023-12-20T10:55:33.364836+0000","last_unstale":"2023-12-20T10:55:34.270671+0000","last_undegraded":"2023-12-20T10:55:34.270671+0000","last_fullsized":"2023-12-20T10:55:34.270671+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.270562+0000","last_change":"2023-12-20T10:55:33.320937+0000","last_active":"2023-12-20T10:55:34.270562+0000","last_peered":"2023-12-20T10:55:34.270562+0000","last_clean":"2023-12-20T10:55:34.270562+0000","last_became_active":"2023-12-20T10:55:33.318837+0000","last_became_peered":"2023-12-20T10:55:33.318837+0000","last_unstale":"2023-12-20T10:55:34.270562+0000","last_undegraded":"2023-12-20T10:55:34.270562+0000","last_fullsized":"2023-12-20T10:55:34.270562+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.272225+0000","last_change":"2023-12-20T10:55:33.310138+0000","last_active":"2023-12-20T10:55:34.272225+0000","last_peered":"2023-12-20T10:55:34.272225+0000","last_clean":"2023-12-20T10:55:34.272225+0000","last_became_active":"2023-12-20T10:55:33.306467+0000","last_became_peered":"2023-12-20T10:55:33.306467+0000","last_unstale":"2023-12-20T10:55:34.272225+0000","last_undegraded":"2023-12-20T10:55:34.272225+0000","last_fullsized":"2023-12-20T10:55:34.272225+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:32.234916+0000","last_clean_scrub_stamp":"2023-12-20T10:55:32.234916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-20T10:55:34.269373+0000","last_change":"2023-12-20T10:55:32.299593+0000","last_active":"2023-12-20T10:55:34.269373+0000","last_peered":"2023-12-20T10:55:34.269373+0000","last_clean":"2023-12-20T10:55:34.269373+0000","last_became_active":"2023-12-20T10:55:32.293211+0000","last_became_peered":"2023-12-20T10:55:32.293211+0000","last_unstale":"2023-12-20T10:55:34.269373+0000","last_undegraded":"2023-12-20T10:55:34.269373+0000","last_fullsized":"2023-12-20T10:55:34.269373+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-20T10:55:31.227209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-20T10:55:31.227209+0000","last_clean_scrub_stamp":"2023-12-20T10:55:31.227209+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26948,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344892,"statfs":{"total":96636764160,"available":96609169408,"internally_reserved":0,"allocated":114688,"data_stored":607364,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29964,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-20T10:55:38.157 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-20T10:55:38.157 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-20T10:55:38.157 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-20T10:55:38.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-20T10:55:38.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:38.265+0000 7f9303ceb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:38.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:55:38.269+0000 7f9303ceb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:55:38.492 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:38.493 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-20T10:55:38.504 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-20T10:55:38.505 INFO:teuthology.run_tasks:Running task cbt... 2023-12-20T10:55:38.518 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi006.front.sepia.ceph.com', 'osds': ['ubuntu@smithi006.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi006.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi006.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi006.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-12-20T10:55:38.519 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-12-20T10:55:38.537 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:55:38.537 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-20T10:55:38.589 DEBUG:teuthology.orchestra.run.smithi006:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-12-20T10:55:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-12-20T10:55:39.037 INFO:tasks.cbt:Installing collectl 2023-12-20T10:55:39.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-12-20T10:55:39.168 INFO:teuthology.orchestra.run.smithi006.stderr:--2023-12-20 10:55:39-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-12-20T10:55:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2023-12-20T10:55:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-12-20T10:55:39.807 INFO:teuthology.orchestra.run.smithi006.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-20T10:55:39.808 INFO:teuthology.orchestra.run.smithi006.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlgsgr4eocfprztRejaDH2ZbOe4YQBdZkD5R1Gsu82T8Np-m0h-WBdQetMKIlUbcfNzVdkzzkJpRdk82e0lmq4bD2bfQ%3D%3D&use_mirror=newcontinuum&r= [following] 2023-12-20T10:55:39.808 INFO:teuthology.orchestra.run.smithi006.stderr:--2023-12-20 10:55:39-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlgsgr4eocfprztRejaDH2ZbOe4YQBdZkD5R1Gsu82T8Np-m0h-WBdQetMKIlUbcfNzVdkzzkJpRdk82e0lmq4bD2bfQ%3D%3D&use_mirror=newcontinuum&r= 2023-12-20T10:55:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-12-20T10:55:39.889 INFO:teuthology.orchestra.run.smithi006.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-12-20T10:55:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-20T10:55:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:Location: https://newcontinuum.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-12-20T10:55:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:--2023-12-20 10:55:40-- https://newcontinuum.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-12-20T10:55:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:Resolving newcontinuum.dl.sourceforge.net (newcontinuum.dl.sourceforge.net)... 64.79.96.4, 2607:ff50:0:11::32 2023-12-20T10:55:40.156 INFO:teuthology.orchestra.run.smithi006.stderr:Connecting to newcontinuum.dl.sourceforge.net (newcontinuum.dl.sourceforge.net)|64.79.96.4|:443... connected. 2023-12-20T10:55:40.255 INFO:teuthology.orchestra.run.smithi006.stderr:HTTP request sent, awaiting response... 200 OK 2023-12-20T10:55:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:Length: 620693 (606K) [application/x-gzip] 2023-12-20T10:55:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:Saving to: ‘collectl.tar.gz’ 2023-12-20T10:55:40.283 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:40.339 INFO:teuthology.orchestra.run.smithi006.stderr: 0K .......... .......... .......... .......... .......... 8% 601K 1s 2023-12-20T10:55:40.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50K .......... .......... .......... .......... .......... 16% 34.5M 0s 2023-12-20T10:55:40.368 INFO:teuthology.orchestra.run.smithi006.stderr: 100K .......... .......... .......... .......... .......... 24% 1.80M 0s 2023-12-20T10:55:40.368 INFO:teuthology.orchestra.run.smithi006.stderr: 150K .......... .......... .......... .......... .......... 32% 55.9M 0s 2023-12-20T10:55:40.395 INFO:teuthology.orchestra.run.smithi006.stderr: 200K .......... .......... .......... .......... .......... 41% 1.85M 0s 2023-12-20T10:55:40.395 INFO:teuthology.orchestra.run.smithi006.stderr: 250K .......... .......... .......... .......... .......... 49% 64.3M 0s 2023-12-20T10:55:40.397 INFO:teuthology.orchestra.run.smithi006.stderr: 300K .......... .......... .......... .......... .......... 57% 35.1M 0s 2023-12-20T10:55:40.398 INFO:teuthology.orchestra.run.smithi006.stderr: 350K .......... .......... .......... .......... .......... 65% 65.7M 0s 2023-12-20T10:55:40.399 INFO:teuthology.orchestra.run.smithi006.stderr: 400K .......... .......... .......... .......... .......... 74% 37.1M 0s 2023-12-20T10:55:40.423 INFO:teuthology.orchestra.run.smithi006.stderr: 450K .......... .......... .......... .......... .......... 82% 2.04M 0s 2023-12-20T10:55:40.423 INFO:teuthology.orchestra.run.smithi006.stderr: 500K .......... .......... .......... .......... .......... 90% 57.7M 0s 2023-12-20T10:55:40.425 INFO:teuthology.orchestra.run.smithi006.stderr: 550K .......... .......... .......... .......... .......... 98% 25.8M 0s 2023-12-20T10:55:40.426 INFO:teuthology.orchestra.run.smithi006.stderr: 600K ...... 100% 182M=0.2s 2023-12-20T10:55:40.426 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:40.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20 10:55:40 (3.48 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-12-20T10:55:40.426 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/ 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/collectl.conf 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/ 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/SlowProc.html 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/WhySummary.html 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Socket.html 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/colmux.html 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Tutorial.html 2023-12-20T10:55:40.466 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Gexpr.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Import.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/style.css 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/FileNaming.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/TheMath.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Environmental.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Process.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/HiResTime.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/slub.jpg 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Lustre.html 2023-12-20T10:55:40.467 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Documentation.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Inodes.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Graphite.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/InputFiles.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Export.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Interrupts.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Startup.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Performance.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Messages.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Logging.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data-detail.html 2023-12-20T10:55:40.468 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/CPUs.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Matrix.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data-brief.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Misc.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Examples.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OpenStack.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data.html 2023-12-20T10:55:40.469 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Exceptions.html 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Playback.html 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Hello.html 2023-12-20T10:55:40.470 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Sockets.html 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Network.html 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Infiniband.html 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Memory.html 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Disks.html 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/service/ 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/service/collectl.service 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/ 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl-suse 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl-debian 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl-generic 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/misc.ph 2023-12-20T10:55:40.471 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/gexpr.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/statsd.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/envrules.std 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/hello.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/vmsum.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/proctree.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/README-WINDOWS 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/formatit.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/colmux 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/INSTALL 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/graphite.ph 2023-12-20T10:55:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/RELEASE-collectl 2023-12-20T10:55:40.474 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/vmstat.ph 2023-12-20T10:55:40.474 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/README 2023-12-20T10:55:40.474 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/man1/ 2023-12-20T10:55:40.474 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/man1/collectl.1 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/man1/colmux.1 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/lexpr.ph 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/client.pl 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/collectl 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/COPYING 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/vnet.ph 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/GPL 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/UNINSTALL 2023-12-20T10:55:40.475 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/ARTISTIC 2023-12-20T10:55:40.529 INFO:teuthology.orchestra.run.smithi006.stderr:./INSTALL: line 90: chkconfig: command not found 2023-12-20T10:55:40.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-12-20T10:55:41.118 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:54 ago on Wed 20 Dec 2023 10:51:47 AM UTC. 2023-12-20T10:55:41.217 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-12-20T10:55:41.219 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-12-20T10:55:41.277 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-20T10:55:41.278 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:55:41.278 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-20T10:55:41.278 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:55:41.278 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-20T10:55:41.279 INFO:teuthology.orchestra.run.smithi006.stdout: librbd-devel x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 23 k 2023-12-20T10:55:41.279 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-12-20T10:55:41.279 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-12-20T10:55:41.279 INFO:teuthology.orchestra.run.smithi006.stdout: perf x86_64 5.14.0-391.el9 appstream 7.6 M 2023-12-20T10:55:41.279 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-20T10:55:41.279 INFO:teuthology.orchestra.run.smithi006.stdout: libradospp-devel x86_64 2:19.0.0-282.ga8ad38c3.el9 ceph 30 k 2023-12-20T10:55:41.280 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:41.280 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-20T10:55:41.280 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-20T10:55:41.280 INFO:teuthology.orchestra.run.smithi006.stdout:Install 5 Packages 2023-12-20T10:55:41.280 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:41.281 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 7.8 M 2023-12-20T10:55:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 11 M 2023-12-20T10:55:41.282 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-20T10:55:41.790 INFO:teuthology.orchestra.run.smithi006.stdout:(1/5): libradospp-devel-19.0.0-282.ga8ad38c3.el 162 kB/s | 30 kB 00:00 2023-12-20T10:55:41.816 INFO:teuthology.orchestra.run.smithi006.stdout:(2/5): librbd-devel-19.0.0-282.ga8ad38c3.el9.x8 110 kB/s | 23 kB 00:00 2023-12-20T10:55:41.916 INFO:teuthology.orchestra.run.smithi006.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 131 kB/s | 13 kB 00:00 2023-12-20T10:55:42.033 INFO:teuthology.orchestra.run.smithi006.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 490 kB/s | 118 kB 00:00 2023-12-20T10:55:43.775 INFO:teuthology.orchestra.run.smithi006.stdout:(5/5): perf-5.14.0-391.el9.x86_64.rpm 3.5 MB/s | 7.6 MB 00:02 2023-12-20T10:55:43.776 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-20T10:55:43.777 INFO:teuthology.orchestra.run.smithi006.stdout:Total 3.1 MB/s | 7.8 MB 00:02 2023-12-20T10:55:43.862 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-20T10:55:43.885 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-20T10:55:43.886 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-20T10:55:44.116 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-20T10:55:44.116 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-20T10:55:44.503 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-20T10:55:44.570 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-12-20T10:55:44.599 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-12-20T10:55:44.630 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradospp-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 3/5 2023-12-20T10:55:44.989 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librbd-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 4/5 2023-12-20T10:55:45.112 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : perf-5.14.0-391.el9.x86_64 5/5 2023-12-20T10:55:46.148 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: perf-5.14.0-391.el9.x86_64 5/5 2023-12-20T10:55:46.148 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradospp-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 1/5 2023-12-20T10:55:46.148 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 2/5 2023-12-20T10:55:46.148 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : perf-5.14.0-391.el9.x86_64 3/5 2023-12-20T10:55:46.148 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: libradospp-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: librbd-devel-2:19.0.0-282.ga8ad38c3.el9.x86_64 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh-2.34-7.el9.x86_64 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-12-20T10:55:46.488 INFO:teuthology.orchestra.run.smithi006.stdout: perf-5.14.0-391.el9.x86_64 2023-12-20T10:55:46.489 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-20T10:55:46.489 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-20T10:55:46.857 INFO:tasks.cbt:benchmark: radosbench 2023-12-20T10:55:46.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-12-20T10:55:46.890 DEBUG:teuthology.orchestra.run.smithi006:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - Settings.cluster: 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'clients': ['ubuntu@smithi006.front.sepia.ceph.com'], 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'head': 'ubuntu@smithi006.front.sepia.ceph.com', 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'is_teuthology': True, 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'iterations': 1, 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'mons': ['ubuntu@smithi006.front.sepia.ceph.com'], 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'osds': ['ubuntu@smithi006.front.sepia.ceph.com'], 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'osds_per_node': 3, 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'pid_dir': '/var/run/ceph', 2023-12-20T10:55:47.314 INFO:teuthology.orchestra.run.smithi006.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'pgp_size': 256, 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'replication': 'replicated'}}, 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'rebuild_every_test': False, 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'rgws': {'ubuntu@smithi006.front.sepia.ceph.com': None}, 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'tmp_dir': '/tmp/cbt', 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'use_existing': True, 2023-12-20T10:55:47.315 INFO:teuthology.orchestra.run.smithi006.stderr: 'user': 'ubuntu'} 2023-12-20T10:55:47.335 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-12-20T10:55:47.336 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:55:47.337 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:55:47.337 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2023-12-20T10:55:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:55:47.340 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-20T10:55:47.342 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:55:47.342 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:55:47.342 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2023-12-20T10:55:47.345 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:55:47.345 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-20T10:55:47.345 INFO:teuthology.orchestra.run.smithi006.stderr:10:55:47 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-20T10:56:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-20T10:56:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.433 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-20T10:56:47.467 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.467 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-12-20T10:56:47.469 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.469 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-12-20T10:56:47.471 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.471 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.471 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2023-12-20T10:56:47.492 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.492 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi006.front.sepia.ceph.com ; done 2023-12-20T10:56:47.496 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.497 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.497 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-12-20T10:56:47.525 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - WARNING - cbt - sudo killall -9 rados 2023-12-20T10:56:47.525 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - WARNING - cbt - error 1 seen, continuing anyway... 2023-12-20T10:56:47.546 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2023-12-20T10:56:47.546 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-12-20T10:56:47.546 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.546 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.546 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-12-20T10:56:47.565 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.565 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.565 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2023-12-20T10:56:47.568 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.568 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-20T10:56:47.569 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.570 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.570 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2023-12-20T10:56:47.573 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.573 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-20T10:56:47.573 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-20T10:56:47.573 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.574 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.574 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-12-20T10:56:47.885 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.885 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:47.885 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-12-20T10:56:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:48 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:48 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:48.708 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-12-20T10:56:49.713 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:49 - INFO - cbt - Checking Health after pool creation. 2023-12-20T10:56:54.718 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:54 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:56:54.718 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:54 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:56:54.718 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-20T10:56:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:10:56:55 - INFO - cbt - Final Pool Health Check. 2023-12-20T10:57:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.078 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-20T10:57:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.398 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-20T10:57:00.412 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.412 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-20T10:57:00.441 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.441 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-12-20T10:57:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.443 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-12-20T10:57:00.444 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.445 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.445 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-12-20T10:57:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:00.560 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-12-20T10:57:01.157 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:01.157 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:01.157 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-12-20T10:57:01.465 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:01.465 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:01.465 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2023-12-20T10:57:01.518 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:01.519 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:01.519 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2023-12-20T10:57:02.265 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.266 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.266 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2023-12-20T10:57:02.268 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.268 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-20T10:57:02.270 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.270 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.270 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - INFO - cbt - Running radosbench write test. 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.273 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2023-12-20T10:57:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:57:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:10:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2023-12-20T10:58:03.396 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.397 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-20T10:58:03.423 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.423 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-20T10:58:03.460 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.460 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2023-12-20T10:58:03.462 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.462 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2023-12-20T10:58:03.463 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.464 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.464 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2023-12-20T10:58:03.779 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.790 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.791 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2023-12-20T10:58:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.811 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi006.front.sepia.ceph.com ; done 2023-12-20T10:58:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - INFO - cbt - Convert results to json format. 2023-12-20T10:58:03.846 INFO:teuthology.orchestra.run.smithi006.stderr:10:58:03 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:03.864 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-12-20T10:58:03.874 DEBUG:teuthology.orchestra.run.smithi006:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-12-20T10:58:04.050 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-12-20T10:58:04.050 INFO:tasks.cbt_performance:reading cbt preferences 2023-12-20T10:58:04.050 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-12-20T10:58:04.118 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi006.front.sepia.ceph.com 2023-12-20T10:58:04.118 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi006.front.sepia.ceph.com 2023-12-20T10:58:04.120 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:58:04.120 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi006.front.sepia.ceph.com of=/dev/stdout 2023-12-20T10:58:04.136 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-20T10:58:04.136 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-12-20T10:58:04.192 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-20T10:58:04.192 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-20T10:58:04.192 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-20T10:58:04.192 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi006.front.sepia.ceph.com of=/dev/stdout 2023-12-20T10:58:04.247 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-20T10:58:04.247 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-12-20T10:58:04.302 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-20T10:58:04.302 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-20T10:58:04.312 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-20T10:58:04.321 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-20T10:58:04.321 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-20T10:58:04.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-20T10:58:04.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:58:04.447+0000 7f8a7322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:58:04.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-20T10:58:04.455+0000 7f8a7322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-20T10:58:04.660 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-20T10:58:04.660 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-20T10:58:04.660 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-20T10:58:04.661 INFO:teuthology.orchestra.run:waiting for 300 2023-12-20T10:58:07.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-20T10:58:10.762 INFO:tasks.ceph.osd.0:Stopped 2023-12-20T10:58:10.762 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-20T10:58:10.763 INFO:teuthology.orchestra.run:waiting for 300 2023-12-20T10:58:13.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-20T10:58:13.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-20T10:58:16.864 INFO:tasks.ceph.osd.1:Stopped 2023-12-20T10:58:16.864 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-20T10:58:16.865 INFO:teuthology.orchestra.run:waiting for 300 2023-12-20T10:58:18.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-20T10:58:18.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-20T10:58:22.966 INFO:tasks.ceph.osd.2:Stopped 2023-12-20T10:58:22.967 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-20T10:58:22.967 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-20T10:58:22.967 INFO:teuthology.orchestra.run:waiting for 300 2023-12-20T10:58:22.999 INFO:tasks.ceph.mgr.x:Stopped 2023-12-20T10:58:22.999 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-20T10:58:22.999 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-20T10:58:22.999 INFO:teuthology.orchestra.run:waiting for 300 2023-12-20T10:58:23.053 INFO:tasks.ceph.mon.a:Stopped 2023-12-20T10:58:23.053 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-20T10:58:23.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-12-20T10:58:23.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:23.140 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-20T10:58:23.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-20T10:58:23.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:23.568 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-20T10:58:23.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:23.657 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-20T10:58:23.714 INFO:tasks.ceph:Archiving mon data... 2023-12-20T10:58:23.715 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823/data/mon.a.tgz 2023-12-20T10:58:23.715 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-12-20T10:58:23.737 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.iOzzBeykdo 2023-12-20T10:58:23.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iOzzBeykdo 2023-12-20T10:58:23.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.iOzzBeykdo 2023-12-20T10:58:23.963 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.iOzzBeykdo is 175KB 2023-12-20T10:58:24.016 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.iOzzBeykdo 2023-12-20T10:58:24.031 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-20T10:58:24.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-20T10:58:24.161 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-20T10:58:24.161 INFO:tasks.ceph:Archiving crash dumps... 2023-12-20T10:58:24.162 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823/remote/smithi006/crash 2023-12-20T10:58:24.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-20T10:58:24.231 INFO:tasks.ceph:Compressing logs... 2023-12-20T10:58:24.231 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-20T10:58:24.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82725.log 2023-12-20T10:58:24.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83900.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82725.log: /var/log/ceph/ceph-client.admin.83900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82957.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.83900.log.gz 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82492.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81295.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83751.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82327.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82809.log 2023-12-20T10:58:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81383.log 2023-12-20T10:58:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82491.log 2023-12-20T10:58:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83835.log 2023-12-20T10:58:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82619.log 2023-12-20T10:58:24.323 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82224.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81761.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82168.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81493.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81533.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82579.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82705.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83812.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81731.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82369.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82789.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83686.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81573.log 2023-12-20T10:58:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82915.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82618.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83899.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82411.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84357.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79936.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82873.log 2023-12-20T10:58:24.325 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-20T10:58:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-20T10:58:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82493.log 2023-12-20T10:58:24.326 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82266.log 2023-12-20T10:58:24.327 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83788.log 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.81295.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.81573.log: /var/log/ceph/ceph-client.admin.82789.log: /var/log/ceph/ceph-client.admin.82493.log: /var/log/ceph/ceph-client.admin.82915.log: /var/log/ceph/ceph.tmp-client.admin.79936.log: /var/log/ceph/ceph-client.admin.82266.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.82873.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.83899.log: /var/log/ceph/ceph-client.admin.84357.log: /var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82725.log.gz 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82957.log.gz 65.6% 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83899.log.gz 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84357.log.gz 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82492.log: /var/log/ceph/ceph-client.admin.82809.log: /var/log/ceph/ceph-client.admin.82327.log: /var/log/ceph/ceph-client.admin.83835.log: /var/log/ceph/ceph-client.admin.82619.log: /var/log/ceph/ceph-client.admin.82491.log: /var/log/ceph/ceph-client.admin.81383.log: /var/log/ceph/ceph-client.admin.83751.log: /var/log/ceph/ceph-client.admin.82168.log: /var/log/ceph/ceph-client.admin.83788.log: /var/log/ceph/ceph-client.admin.83812.log: /var/log/ceph/ceph-client.admin.82224.log: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.81761.log: /var/log/ceph/ceph-client.admin.81533.log: /var/log/ceph/ceph-client.admin.82705.log: /var/log/ceph/ceph-client.admin.82369.log: /var/log/ceph/ceph-client.admin.81493.log: /var/log/ceph/ceph-client.admin.82579.log: /var/log/ceph/ceph-client.admin.82618.log: /var/log/ceph/ceph-client.admin.81731.log: /var/log/ceph/ceph-client.admin.83686.log: /var/log/ceph/ceph-client.admin.82411.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82789.log.gz 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82493.log.gz 2023-12-20T10:58:24.334 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82915.log.gz 2023-12-20T10:58:24.338 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82873.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82266.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82492.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82327.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82809.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83835.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82619.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82491.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81383.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83751.log.gz 2023-12-20T10:58:24.340 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82168.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83812.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82224.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82705.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82369.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82411.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82579.log.gz -- replaced with /var/log/ceph/ceph-client.admin.82618.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83686.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83788.log.gz 2023-12-20T10:58:24.341 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81573.log.gz 2023-12-20T10:58:24.343 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81731.log.gz 2023-12-20T10:58:24.344 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81761.log.gz 2023-12-20T10:58:24.348 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81493.log.gz 2023-12-20T10:58:24.348 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81533.log.gz 2023-12-20T10:58:24.352 INFO:teuthology.orchestra.run.smithi006.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79936.log.gz 2023-12-20T10:58:24.357 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81295.log.gz 2023-12-20T10:58:24.357 INFO:teuthology.orchestra.run.smithi006.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-20T10:58:24.427 INFO:teuthology.orchestra.run.smithi006.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-20T10:58:24.429 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-20T10:58:26.225 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-20T10:58:26.354 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-20T10:58:26.416 INFO:teuthology.orchestra.run.smithi006.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-20T10:58:26.418 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-20T10:58:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m2.131s 2023-12-20T10:58:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m3.826s 2023-12-20T10:58:26.418 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.174s 2023-12-20T10:58:26.418 INFO:tasks.ceph:Archiving logs... 2023-12-20T10:58:26.418 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823/remote/smithi006/log 2023-12-20T10:58:26.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-20T10:58:27.110 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-20T10:58:27.120 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-20T10:58:27.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-20T10:58:27.179 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-20T10:58:27.188 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-20T10:58:27.190 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-20T10:58:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2023-12-20T10:58:27.271 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-20T10:58:27.271 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2023-12-20T10:58:27.272 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +1127us[+1132us] +/- 79ms 2023-12-20T10:58:27.272 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 +98us[ +103us] +/- 76ms 2023-12-20T10:58:27.272 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -1315us[-1310us] +/- 85ms 2023-12-20T10:58:27.272 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-20T10:58:27.284 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-20T10:58:27.293 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-20T10:58:27.294 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-20T10:58:27.313 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-20T10:58:27.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-20T10:58:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-20T10:58:27.522 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-20T10:58:27.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-20T10:58:27.580 INFO:teuthology.task.internal:Duration was 778.363645 seconds 2023-12-20T10:58:27.580 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-20T10:58:27.597 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-20T10:58:27.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-20T10:58:27.738 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-20T10:58:28.140 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-20T10:58:28.141 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2023-12-20T10:58:28.141 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-20T10:58:28.163 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-20T10:58:28.164 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-20T10:58:28.237 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-20T10:58:28.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-20T10:58:28.384 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-20T10:58:28.394 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-20T10:58:28.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-20T10:58:28.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-20T10:58:28.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-20T10:58:28.524 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2023-12-20T10:58:28.548 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-20T10:58:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-20T10:58:28.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-20T10:58:28.572 INFO:teuthology.task.internal:Transferring archived files... 2023-12-20T10:58:28.573 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-12-20_10:24:11-crimson-rados-wip-matanb-crimson-only-submit-error-cleanup-distro-crimson-smithi/7497823/remote/smithi006 2023-12-20T10:58:28.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-20T10:58:28.668 INFO:teuthology.task.internal:Removing archive directory... 2023-12-20T10:58:28.668 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-20T10:58:28.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-20T10:58:28.738 INFO:teuthology.task.internal:Not uploading archives. 2023-12-20T10:58:28.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-20T10:58:28.755 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-20T10:58:28.756 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-20T10:58:28.779 INFO:teuthology.orchestra.run.smithi006.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 20 10:58 /home/ubuntu/cephtest 2023-12-20T10:58:28.780 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-20T10:58:28.790 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-20T10:58:28.824 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-only-submit-error-cleanup&var-job_id_selected=7497823 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 778.3636445999146 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-12-20T10:58:28.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-20T10:58:28.899 INFO:teuthology.run:pass