2023-10-31T23:16:59.950 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-31T23:16:59.951 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530 --owner scheduled_matan@teuthology --name matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} --verbose -- /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530/orig.config.yaml 2023-10-31T23:16:59.995 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-31T23:17:00.070 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530 branch: wip-matanb-crimson-do_osd_ops_execute-part3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} email: null first_in_suite: false job_id: '7442530' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-do_osd_ops_execute-part3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: a9edd23b5a8ddacab225824948b06b370baf276b owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6029 sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: a9edd23b5a8ddacab225824948b06b370baf276b targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsgaC1yCuZUf5QYeSnmN1ltMdYBBt5S9+cR1tpCFj9yPqIVSYa6vAS/p0ZO964kV4eGYOEkhTu+Nl4PFeTXpx0= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-31_22:58:07 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-31T23:17:00.071 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa; will attempt to use it 2023-10-31T23:17:00.072 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa/tasks 2023-10-31T23:17:00.072 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-31T23:17:00.074 INFO:teuthology.task.internal:Checking packages... 2023-10-31T23:17:00.096 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2' 2023-10-31T23:17:00.097 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-31T23:17:00.097 INFO:teuthology.packaging:ref: None 2023-10-31T23:17:00.098 INFO:teuthology.packaging:tag: None 2023-10-31T23:17:00.098 INFO:teuthology.packaging:branch: wip-matanb-crimson-do_osd_ops_execute-part3 2023-10-31T23:17:00.098 INFO:teuthology.packaging:sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 2023-10-31T23:17:00.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-do_osd_ops_execute-part3 2023-10-31T23:17:00.240 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6795.gd1dcabba 2023-10-31T23:17:00.241 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-31T23:17:00.250 INFO:teuthology.task.internal:no buildpackages task found 2023-10-31T23:17:00.250 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-31T23:17:00.255 INFO:teuthology.task.internal:Saving configuration 2023-10-31T23:17:00.271 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-31T23:17:00.276 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-31T23:17:00.298 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-31 23:12:19.961338', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsgaC1yCuZUf5QYeSnmN1ltMdYBBt5S9+cR1tpCFj9yPqIVSYa6vAS/p0ZO964kV4eGYOEkhTu+Nl4PFeTXpx0='} 2023-10-31T23:17:00.298 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-31T23:17:00.304 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-31T23:17:00.305 INFO:teuthology.run_tasks:Running task console_log... 2023-10-31T23:17:00.344 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9d3c565940>, signals=[15]) 2023-10-31T23:17:00.345 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-31T23:17:00.351 INFO:teuthology.task.internal:Opening connections... 2023-10-31T23:17:00.352 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:17:00.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T23:17:00.452 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-31T23:17:00.458 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-10-31T23:17:00.522 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-10-31T23:17:00.523 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-10-31T23:17:00.579 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2023-10-31T23:17:00.579 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2023-10-31T23:17:00.579 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2023-10-31T23:17:00.580 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2023-10-31T23:17:00.580 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2023-10-31T23:17:00.580 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2023-10-31T23:17:00.580 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-31T23:17:00.580 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2023-10-31T23:17:00.581 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2023-10-31T23:17:00.581 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-31T23:17:00.581 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2023-10-31T23:17:00.581 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-31T23:17:00.581 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-31T23:17:00.582 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-31T23:17:00.582 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-10-31T23:17:00.602 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-31T23:17:00.610 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-31T23:17:00.639 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-31T23:17:00.640 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-10-31T23:17:00.656 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-31T23:17:00.661 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-31T23:17:00.661 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-10-31T23:17:00.719 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-31T23:17:00.751 INFO:teuthology.run_tasks:Running task kernel... 2023-10-31T23:17:00.775 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-31T23:17:00.776 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-31T23:17:00.776 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-31T23:17:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T23:17:00.793 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-10-31T23:17:00.851 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-361.el9.x86_64 2023-10-31T23:17:00.851 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-361.el9.x86_64 2023-10-31T23:17:00.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2023-10-31T23:17:02.942 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-31T23:17:04.276 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2023-10-31T23:17:06.233 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 7.9 MB 00:01 2023-10-31T23:17:11.151 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 18 MB 00:03 2023-10-31T23:17:15.260 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-10-31T23:17:16.860 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2023-10-31T23:17:18.795 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-10-31T23:17:23.420 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-10-31T23:17:25.560 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-31T23:17:25.560 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-31T23:17:25.657 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:17:25.664 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:17:25.664 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-31T23:17:25.664 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:17:25.664 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:17:25.665 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-31T23:17:25.665 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:17:25.665 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-31T23:17:25.665 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-31T23:17:25.665 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-31T23:17:25.666 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:17:25.666 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:17:25.666 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:17:25.666 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2023-10-31T23:17:25.666 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:17:25.667 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 95 M 2023-10-31T23:17:25.667 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 120 M 2023-10-31T23:17:25.667 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:17:27.221 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 3.6 MB/s | 5.3 MB 00:01 2023-10-31T23:17:27.471 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 11 MB/s | 20 MB 00:01 2023-10-31T23:17:28.805 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 12 MB/s | 38 MB 00:03 2023-10-31T23:17:34.854 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-modules-core-5.14.0-378.el9.x86_6 4.2 MB/s | 32 MB 00:07 2023-10-31T23:17:34.855 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:17:34.855 INFO:teuthology.orchestra.run.smithi050.stdout:Total 10 MB/s | 95 MB 00:09 2023-10-31T23:17:35.734 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:17:36.162 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:17:36.162 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:17:37.478 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:17:37.479 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:17:40.248 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:17:40.897 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-31T23:17:41.022 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T23:17:43.944 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T23:17:44.127 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T23:17:49.110 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T23:17:49.243 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-31T23:17:54.251 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T23:18:24.727 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T23:18:24.773 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-31T23:18:25.209 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-31T23:18:25.210 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-31T23:18:25.210 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T23:18:25.210 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T23:18:25.654 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T23:18:25.655 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:18:25.655 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:18:25.655 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-31T23:18:25.655 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-31T23:18:25.656 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-31T23:18:25.656 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-31T23:18:25.656 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:18:25.656 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:18:26.174 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-10-31T23:18:26.360 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-31T23:18:26.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-31T23:18:26.360 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-31T23:18:26.361 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-31T23:18:26.361 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-31T23:18:26.361 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-31T23:18:26.362 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2023-10-31T23:18:26.362 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-10-31T23:18:26.486 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-31T23:18:26.487 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-31T23:18:26.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2023-10-31T23:18:26.534 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2023-10-31T23:18:26.534 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2023-10-31T23:18:26.535 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2023-10-31T23:18:26.535 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 68.el9 2023-10-31T23:18:26.535 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2023-10-31T23:18:26.535 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-31T23:18:26.535 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2023-10-31T23:18:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8264254 2023-10-31T23:18:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2023-10-31T23:18:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-31T23:18:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-31T23:18:26.536 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-31T23:18:26.537 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-31T23:18:26.537 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2023-10-31T23:18:26.537 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2023-10-31T23:18:26.537 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-31T23:18:26.537 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2023-10-31T23:18:26.537 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2023-10-31T23:18:26.538 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:18:26.538 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-31T23:18:26.538 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-31T23:18:26.538 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-31T23:18:26.538 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2023-10-31T23:18:26.539 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:18:26.539 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2023-10-31T23:18:26.540 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-31T23:18:26.540 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-378.el9.x86_64 2023-10-31T23:18:26.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-31T23:18:27.140 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-10-31T23:18:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-31T23:18:28.562 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-10-31T23:18:28.565 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-10-31T23:18:28.581 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.mudhX8tro9 2023-10-31T23:18:28.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mudhX8tro9 2023-10-31T23:18:28.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.mudhX8tro9 2023-10-31T23:18:28.793 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.mudhX8tro9 is 6KB 2023-10-31T23:18:28.842 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.mudhX8tro9 2023-10-31T23:18:28.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2023-10-31T23:18:28.935 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-31T23:18:28.935 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-31T23:18:28.935 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-31T23:18:28.935 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-31T23:18:28.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-31T23:18:29.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2023-10-31T23:18:59.130 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-31T23:18:59.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-31T23:18:59.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T23:19:59.186 DEBUG:teuthology.orchestra.remote:timed out 2023-10-31T23:20:08.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-31T23:20:08.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T23:20:08.630 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-10-31T23:20:08.985 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-31T23:20:08.986 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-31T23:20:08.986 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-10-31T23:20:09.001 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-378.el9.x86_64 2023-10-31T23:20:09.001 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-31T23:20:09.001 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-31T23:20:09.002 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-31T23:20:10.003 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-31T23:20:10.005 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-31T23:20:10.049 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-10-31T23:20:10.091 DEBUG:teuthology.parallel:result is None 2023-10-31T23:20:10.091 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-31T23:20:10.097 INFO:teuthology.task.internal:Creating test directory... 2023-10-31T23:20:10.098 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-31T23:20:10.113 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-31T23:20:10.120 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-31T23:20:10.127 INFO:teuthology.task.internal:Creating archive directory... 2023-10-31T23:20:10.127 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-31T23:20:10.198 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-31T23:20:10.205 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-31T23:20:10.205 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-31T23:20:10.240 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-31T23:20:10.256 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-31T23:20:10.258 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-31T23:20:10.265 INFO:teuthology.task.internal:Configuring sudo... 2023-10-31T23:20:10.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-31T23:20:10.326 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-31T23:20:10.336 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-31T23:20:10.336 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-31T23:20:10.378 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-31T23:20:10.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-31T23:20:10.507 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-31T23:20:10.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-31T23:20:10.630 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:20:10.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-31T23:20:10.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-10-31T23:20:10.785 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-31T23:20:10.875 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-31T23:20:10.883 INFO:teuthology.task.internal:Starting timer... 2023-10-31T23:20:10.884 INFO:teuthology.run_tasks:Running task pcp... 2023-10-31T23:20:10.893 INFO:teuthology.run_tasks:Running task selinux... 2023-10-31T23:20:10.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2023-10-31T23:20:10.968 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2023-10-31T23:20:10.970 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-31T23:20:10.970 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2023-10-31T23:20:10.998 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2023-10-31T23:20:10.998 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive'} 2023-10-31T23:20:10.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-31T23:20:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T23:20:11.067 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-31T23:20:11.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2023-10-31T23:20:11.142 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-31T23:20:11.151 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-31T23:20:11.151 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-31T23:20:11.167 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-31T23:20:11.169 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-31T23:25:44.320 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com')] 2023-10-31T23:25:44.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-31T23:25:44.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T23:25:44.419 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-10-31T23:25:44.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-10-31T23:25:44.490 INFO:teuthology.run_tasks:Running task clock... 2023-10-31T23:25:44.504 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-31T23:25:44.504 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-31T23:25:44.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-31T23:25:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-31T23:25:44.583 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-31T23:25:44.620 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2023-10-31T23:25:44.638 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2023-10-31T23:25:44.659 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-31T23:25:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-31T23:25:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2023-10-31T23:25:44.729 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-31T23:25:44.729 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-10-31T23:25:44.729 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T23:25:44.729 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T23:25:44.729 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T23:25:44.730 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T23:25:44.731 INFO:teuthology.run_tasks:Running task install... 2023-10-31T23:25:44.740 DEBUG:teuthology.task.install:project ceph 2023-10-31T23:25:44.740 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'}} 2023-10-31T23:25:44.741 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'} 2023-10-31T23:25:44.741 INFO:teuthology.task.install:Using flavor: crimson 2023-10-31T23:25:44.747 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-31T23:25:44.747 INFO:teuthology.task.install:extra packages: [] 2023-10-31T23:25:44.747 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2', 'tag': None, 'wait_for_package': False} 2023-10-31T23:25:44.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 2023-10-31T23:25:44.911 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/ 2023-10-31T23:25:44.912 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6795.gd1dcabba 2023-10-31T23:25:45.021 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-31T23:25:45.022 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:25:45.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-31T23:25:45.054 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-31T23:25:45.054 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-31T23:25:45.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-31T23:25:45.213 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-31T23:25:45.243 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2023-10-31T23:25:45.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2023-10-31T23:25:45.510 INFO:teuthology.orchestra.run.smithi050.stdout:72 files removed 2023-10-31T23:25:45.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2023-10-31T23:25:46.218 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 236 kB/s | 81 kB 00:00 2023-10-31T23:25:46.569 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 32 kB/s | 9.5 kB 00:00 2023-10-31T23:25:46.928 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2023-10-31T23:25:47.289 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - AppStream 20 MB/s | 6.3 MB 00:00 2023-10-31T23:25:48.697 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2023-10-31T23:25:50.421 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 7.9 MB 00:01 2023-10-31T23:25:53.740 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 8.0 MB/s | 18 MB 00:02 2023-10-31T23:25:58.374 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 5.8 MB 00:01 2023-10-31T23:26:00.025 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-10-31T23:26:05.338 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 19 MB 00:05 2023-10-31T23:26:10.176 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-10-31T23:26:12.666 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:26:12.669 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:26:12.669 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:26:12.669 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:26:12.670 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:26:12.670 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 8.4 M 2023-10-31T23:26:12.670 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2023-10-31T23:26:12.670 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 3.1 M 2023-10-31T23:26:12.670 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 3.0 M 2023-10-31T23:26:12.671 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:26:12.671 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options 2023-10-31T23:26:12.671 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-31T23:26:12.671 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 6.6 M 2023-10-31T23:26:12.671 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 17 M 2023-10-31T23:26:12.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 24 k 2023-10-31T23:26:12.672 INFO:teuthology.orchestra.run.smithi050.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-31T23:26:12.672 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-31T23:26:12.672 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-31T23:26:12.672 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-31T23:26:12.672 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 727 k 2023-10-31T23:26:12.673 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-31T23:26:12.673 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-31T23:26:12.673 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-31T23:26:12.673 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 469 k 2023-10-31T23:26:12.673 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-31T23:26:12.674 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 4.5 M 2023-10-31T23:26:12.674 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-31T23:26:12.674 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-31T23:26:12.674 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-31T23:26:12.674 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse 2023-10-31T23:26:12.675 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 45 k 2023-10-31T23:26:12.675 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 127 k 2023-10-31T23:26:12.675 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 162 k 2023-10-31T23:26:12.675 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-31T23:26:12.675 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 321 k 2023-10-31T23:26:12.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 298 k 2023-10-31T23:26:12.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 99 k 2023-10-31T23:26:12.676 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-31T23:26:12.676 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-31T23:26:12.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:12.677 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:26:12.677 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:26:12.677 INFO:teuthology.orchestra.run.smithi050.stdout:Install 28 Packages 2023-10-31T23:26:12.677 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2023-10-31T23:26:12.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:12.678 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 54 M 2023-10-31T23:26:12.678 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:26:15.794 INFO:teuthology.orchestra.run.smithi050.stdout:(1/30): ceph-radosgw-18.0.0-6795.gd1dcabba.el9. 2.8 MB/s | 8.4 MB 00:03 2023-10-31T23:26:15.861 INFO:teuthology.orchestra.run.smithi050.stdout:(2/30): ceph-selinux-18.0.0-6795.gd1dcabba.el9. 362 kB/s | 24 kB 00:00 2023-10-31T23:26:16.019 INFO:teuthology.orchestra.run.smithi050.stdout:(3/30): libcephfs2-18.0.0-6795.gd1dcabba.el9.x8 4.5 MB/s | 727 kB 00:00 2023-10-31T23:26:16.111 INFO:teuthology.orchestra.run.smithi050.stdout:(4/30): libradosstriper1-18.0.0-6795.gd1dcabba. 5.0 MB/s | 469 kB 00:00 2023-10-31T23:26:16.612 INFO:teuthology.orchestra.run.smithi050.stdout:(5/30): ceph-base-18.0.0-6795.gd1dcabba.el9.x86 1.7 MB/s | 6.6 MB 00:03 2023-10-31T23:26:16.679 INFO:teuthology.orchestra.run.smithi050.stdout:(6/30): librgw2-18.0.0-6795.gd1dcabba.el9.x86_6 7.9 MB/s | 4.5 MB 00:00 2023-10-31T23:26:16.704 INFO:teuthology.orchestra.run.smithi050.stdout:(7/30): python3-ceph-argparse-18.0.0-6795.gd1dc 491 kB/s | 45 kB 00:00 2023-10-31T23:26:16.746 INFO:teuthology.orchestra.run.smithi050.stdout:(8/30): python3-ceph-common-18.0.0-6795.gd1dcab 1.9 MB/s | 127 kB 00:00 2023-10-31T23:26:16.805 INFO:teuthology.orchestra.run.smithi050.stdout:(9/30): python3-cephfs-18.0.0-6795.gd1dcabba.el 1.6 MB/s | 162 kB 00:00 2023-10-31T23:26:16.830 INFO:teuthology.orchestra.run.smithi050.stdout:(10/30): python3-rados-18.0.0-6795.gd1dcabba.el 3.8 MB/s | 321 kB 00:00 2023-10-31T23:26:16.897 INFO:teuthology.orchestra.run.smithi050.stdout:(11/30): python3-rgw-18.0.0-6795.gd1dcabba.el9. 1.5 MB/s | 99 kB 00:00 2023-10-31T23:26:16.963 INFO:teuthology.orchestra.run.smithi050.stdout:(12/30): python3-rbd-18.0.0-6795.gd1dcabba.el9. 1.8 MB/s | 298 kB 00:00 2023-10-31T23:26:17.006 INFO:teuthology.orchestra.run.smithi050.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 975 kB/s | 106 kB 00:00 2023-10-31T23:26:17.031 INFO:teuthology.orchestra.run.smithi050.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2023-10-31T23:26:17.073 INFO:teuthology.orchestra.run.smithi050.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-10-31T23:26:17.106 INFO:teuthology.orchestra.run.smithi050.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-10-31T23:26:17.132 INFO:teuthology.orchestra.run.smithi050.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 710 kB/s | 41 kB 00:00 2023-10-31T23:26:17.257 INFO:teuthology.orchestra.run.smithi050.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 301 kB/s | 45 kB 00:00 2023-10-31T23:26:17.641 INFO:teuthology.orchestra.run.smithi050.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 218 kB/s | 111 kB 00:00 2023-10-31T23:26:17.874 INFO:teuthology.orchestra.run.smithi050.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 501 kB/s | 309 kB 00:00 2023-10-31T23:26:18.100 INFO:teuthology.orchestra.run.smithi050.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 108 kB/s | 24 kB 00:00 2023-10-31T23:26:18.208 INFO:teuthology.orchestra.run.smithi050.stdout:(22/30): libarrow-9.0.0-9.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2023-10-31T23:26:18.225 INFO:teuthology.orchestra.run.smithi050.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 392 kB/s | 49 kB 00:00 2023-10-31T23:26:18.309 INFO:teuthology.orchestra.run.smithi050.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2023-10-31T23:26:18.342 INFO:teuthology.orchestra.run.smithi050.stdout:(25/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 7.0 MB/s | 837 kB 00:00 2023-10-31T23:26:18.418 INFO:teuthology.orchestra.run.smithi050.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-10-31T23:26:18.476 INFO:teuthology.orchestra.run.smithi050.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2023-10-31T23:26:18.810 INFO:teuthology.orchestra.run.smithi050.stdout:(28/30): librbd1-18.0.0-6795.gd1dcabba.el9.x86_ 9.1 MB/s | 3.0 MB 00:00 2023-10-31T23:26:19.111 INFO:teuthology.orchestra.run.smithi050.stdout:(29/30): librados2-18.0.0-6795.gd1dcabba.el9.x8 4.5 MB/s | 3.1 MB 00:00 2023-10-31T23:26:19.944 INFO:teuthology.orchestra.run.smithi050.stdout:(30/30): ceph-common-18.0.0-6795.gd1dcabba.el9. 2.4 MB/s | 17 MB 00:07 2023-10-31T23:26:19.950 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:26:19.950 INFO:teuthology.orchestra.run.smithi050.stdout:Total 7.4 MB/s | 54 MB 00:07 2023-10-31T23:26:19.963 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:26:20.006 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:26:20.006 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:26:20.508 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:26:20.509 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:26:21.933 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:26:22.310 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-31T23:26:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/32 2023-10-31T23:26:22.469 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/32 2023-10-31T23:26:22.563 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/32 2023-10-31T23:26:22.589 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/32 2023-10-31T23:26:22.636 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/32 2023-10-31T23:26:22.684 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-31T23:26:22.711 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-31T23:26:22.740 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-31T23:26:22.833 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:18.0.0-6795.gd1dcabba.el9. 8/32 2023-10-31T23:26:22.953 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 9/32 2023-10-31T23:26:22.976 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_6 10/32 2023-10-31T23:26:23.075 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_6 10/32 2023-10-31T23:26:23.109 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-31T23:26:23.221 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-31T23:26:23.252 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-31T23:26:23.293 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-31T23:26:23.324 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-31T23:26:23.354 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-31T23:26:23.391 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-31T23:26:23.415 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-31T23:26:23.760 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-31T23:26:23.787 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 20/32 2023-10-31T23:26:23.878 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 20/32 2023-10-31T23:26:23.908 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 21/32 2023-10-31T23:26:23.945 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-31T23:26:24.402 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-31T23:26:24.431 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 24/32 2023-10-31T23:26:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 24/32 2023-10-31T23:26:24.565 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 25/32 2023-10-31T23:26:24.600 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:18.0.0-6795.gd1dcabba.el9.x8 26/32 2023-10-31T23:26:26.713 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 27/32 2023-10-31T23:26:26.747 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 27/32 2023-10-31T23:26:27.539 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 27/32 2023-10-31T23:26:27.590 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 28/32 2023-10-31T23:26:27.671 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 28/32 2023-10-31T23:26:27.671 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-31T23:26:27.672 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-31T23:26:27.672 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:27.707 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/32 2023-10-31T23:26:38.013 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/32 2023-10-31T23:26:38.013 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2023-10-31T23:26:38.013 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2023-10-31T23:26:38.013 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2023-10-31T23:26:38.014 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2023-10-31T23:26:38.014 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2023-10-31T23:26:38.014 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2023-10-31T23:26:38.014 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2023-10-31T23:26:38.014 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:38.889 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/32 2023-10-31T23:26:38.916 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/32 2023-10-31T23:26:38.916 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:26:38.916 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-31T23:26:38.917 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-31T23:26:38.917 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-31T23:26:38.917 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:38.918 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-31T23:26:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-31T23:26:39.008 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T23:26:40.192 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T23:26:40.192 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/32 2023-10-31T23:26:40.192 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/32 2023-10-31T23:26:40.192 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/32 2023-10-31T23:26:40.192 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/32 2023-10-31T23:26:40.193 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 5/32 2023-10-31T23:26:40.193 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_6 6/32 2023-10-31T23:26:40.193 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 7/32 2023-10-31T23:26:40.193 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6795.gd1dcabba.el9. 8/32 2023-10-31T23:26:40.193 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:18.0.0-6795.gd1dcabba.el9.x8 9/32 2023-10-31T23:26:40.194 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 10/32 2023-10-31T23:26:40.194 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 11/32 2023-10-31T23:26:40.194 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 12/32 2023-10-31T23:26:40.194 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 13/32 2023-10-31T23:26:40.194 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-31T23:26:40.195 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-31T23:26:40.195 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-31T23:26:40.195 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-31T23:26:40.195 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-31T23:26:40.195 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-31T23:26:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-31T23:26:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-31T23:26:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-31T23:26:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-31T23:26:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-31T23:26:40.196 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-31T23:26:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-31T23:26:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-31T23:26:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-31T23:26:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/32 2023-10-31T23:26:40.197 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-31T23:26:40.198 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 31/32 2023-10-31T23:26:41.292 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T23:26:41.292 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:41.293 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2023-10-31T23:26:41.293 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.293 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.293 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:26:41.293 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-31T23:26:41.294 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.294 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.294 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.294 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-31T23:26:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-31T23:26:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-31T23:26:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-31T23:26:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.296 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-31T23:26:41.296 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-31T23:26:41.296 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-31T23:26:41.296 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.296 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-31T23:26:41.297 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.297 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-31T23:26:41.297 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-31T23:26:41.297 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-31T23:26:41.297 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.298 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.298 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.298 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-31T23:26:41.298 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.298 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.299 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:26:41.299 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2023-10-31T23:26:41.299 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-31T23:26:41.299 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:41.299 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:26:41.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2023-10-31T23:26:42.203 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:32 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:26:42.351 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:26:42.352 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:26:42.352 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:26:42.352 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:26:42.352 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:26:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 117 M 2023-10-31T23:26:42.353 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:26:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-31T23:26:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-31T23:26:42.353 INFO:teuthology.orchestra.run.smithi050.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-31T23:26:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 161 k 2023-10-31T23:26:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-31T23:26:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-31T23:26:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-31T23:26:42.354 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-31T23:26:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-31T23:26:42.355 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:42.355 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:26:42.355 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:26:42.355 INFO:teuthology.orchestra.run.smithi050.stdout:Install 10 Packages 2023-10-31T23:26:42.356 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:26:42.356 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 120 M 2023-10-31T23:26:42.357 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.1 G 2023-10-31T23:26:42.357 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:26:42.580 INFO:teuthology.orchestra.run.smithi050.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 3.2 MB/s | 418 kB 00:00 2023-10-31T23:26:42.672 INFO:teuthology.orchestra.run.smithi050.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.1 MB/s | 186 kB 00:00 2023-10-31T23:26:42.706 INFO:teuthology.orchestra.run.smithi050.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.6 MB/s | 217 kB 00:00 2023-10-31T23:26:42.748 INFO:teuthology.orchestra.run.smithi050.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2023-10-31T23:26:42.781 INFO:teuthology.orchestra.run.smithi050.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-31T23:26:42.990 INFO:teuthology.orchestra.run.smithi050.stdout:(6/10): libcephsqlite-18.0.0-6795.gd1dcabba.el9 298 kB/s | 161 kB 00:00 2023-10-31T23:26:43.015 INFO:teuthology.orchestra.run.smithi050.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 800 kB/s | 187 kB 00:00 2023-10-31T23:26:43.315 INFO:teuthology.orchestra.run.smithi050.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 406 kB/s | 122 kB 00:00 2023-10-31T23:26:43.466 INFO:teuthology.orchestra.run.smithi050.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.6 MB/s | 1.3 MB 00:00 2023-10-31T23:26:50.899 INFO:teuthology.orchestra.run.smithi050.stdout:(10/10): ceph-test-18.0.0-6795.gd1dcabba.el9.x8 14 MB/s | 117 MB 00:08 2023-10-31T23:26:50.902 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:26:50.902 INFO:teuthology.orchestra.run.smithi050.stdout:Total 14 MB/s | 120 MB 00:08 2023-10-31T23:26:50.917 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:26:50.941 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:26:50.941 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:26:51.798 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:26:51.799 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:26:52.945 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:26:53.025 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-31T23:26:53.063 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-31T23:26:53.120 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-31T23:26:53.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-31T23:26:53.203 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-31T23:26:53.231 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-31T23:26:53.271 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-31T23:26:53.311 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-31T23:26:53.337 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 9/10 2023-10-31T23:27:10.480 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 9/10 2023-10-31T23:27:10.554 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 10/10 2023-10-31T23:27:18.835 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 10/10 2023-10-31T23:27:18.836 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/10 2023-10-31T23:27:18.836 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/10 2023-10-31T23:27:18.836 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-31T23:27:18.836 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-31T23:27:18.837 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-31T23:27:18.837 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-31T23:27:18.837 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-31T23:27:18.837 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-31T23:27:18.837 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-31T23:27:19.273 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-31T23:27:19.273 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:19.274 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:27:19.274 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:27:19.274 INFO:teuthology.orchestra.run.smithi050.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-31T23:27:19.274 INFO:teuthology.orchestra.run.smithi050.stdout: jq-1.6-15.el9.x86_64 2023-10-31T23:27:19.275 INFO:teuthology.orchestra.run.smithi050.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-31T23:27:19.275 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:27:19.275 INFO:teuthology.orchestra.run.smithi050.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-31T23:27:19.275 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-31T23:27:19.275 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-31T23:27:19.276 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-31T23:27:19.276 INFO:teuthology.orchestra.run.smithi050.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-31T23:27:19.276 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:19.276 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:27:19.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2023-10-31T23:27:20.202 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:10 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:27:20.351 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:27:20.353 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2023-10-31T23:27:20.353 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:27:20.353 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2023-10-31T23:27:20.354 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:27:20.354 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 6.5 k 2023-10-31T23:27:20.354 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:27:20.354 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 2.0 M 2023-10-31T23:27:20.354 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 1.5 M 2023-10-31T23:27:20.355 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 243 k 2023-10-31T23:27:20.355 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 8.8 M 2023-10-31T23:27:20.355 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 117 M 2023-10-31T23:27:20.355 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-31T23:27:20.355 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-31T23:27:20.356 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-31T23:27:20.356 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-31T23:27:20.356 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-31T23:27:20.357 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-31T23:27:20.357 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-31T23:27:20.357 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-31T23:27:20.357 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-31T23:27:20.357 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-31T23:27:20.358 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-31T23:27:20.358 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-31T23:27:20.358 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-31T23:27:20.358 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-31T23:27:20.358 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-31T23:27:20.359 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-31T23:27:20.359 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-31T23:27:20.359 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-31T23:27:20.359 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-31T23:27:20.359 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-31T23:27:20.359 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-31T23:27:20.360 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-31T23:27:20.360 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-31T23:27:20.360 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-31T23:27:20.360 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-31T23:27:20.360 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-31T23:27:20.361 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-31T23:27:20.361 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:20.361 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:27:20.361 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2023-10-31T23:27:20.361 INFO:teuthology.orchestra.run.smithi050.stdout:Install 33 Packages 2023-10-31T23:27:20.362 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:20.362 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 133 M 2023-10-31T23:27:20.363 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.4 G 2023-10-31T23:27:20.363 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:27:20.735 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-18.0.0-6795.gd1dcabba.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2023-10-31T23:27:21.010 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-mds-18.0.0-6795.gd1dcabba.el9.x86_ 4.5 MB/s | 2.0 MB 00:00 2023-10-31T23:27:21.077 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): ceph-mgr-18.0.0-6795.gd1dcabba.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2023-10-31T23:27:21.186 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6795.gd1dc 2.2 MB/s | 243 kB 00:00 2023-10-31T23:27:21.278 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2023-10-31T23:27:21.311 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2023-10-31T23:27:21.345 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-31T23:27:21.387 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-10-31T23:27:21.429 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-10-31T23:27:21.463 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-10-31T23:27:21.496 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-31T23:27:21.530 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-10-31T23:27:21.905 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): ceph-mon-18.0.0-6795.gd1dcabba.el9.x86 7.5 MB/s | 8.8 MB 00:01 2023-10-31T23:27:21.947 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2023-10-31T23:27:22.247 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2023-10-31T23:27:22.273 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 134 kB/s | 43 kB 00:00 2023-10-31T23:27:22.456 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 824 kB/s | 172 kB 00:00 2023-10-31T23:27:22.557 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2023-10-31T23:27:22.649 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2023-10-31T23:27:22.741 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 253 kB/s | 23 kB 00:00 2023-10-31T23:27:22.816 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 661 kB/s | 358 kB 00:00 2023-10-31T23:27:22.841 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2023-10-31T23:27:22.917 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2023-10-31T23:27:22.941 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2023-10-31T23:27:23.017 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 787 kB/s | 79 kB 00:00 2023-10-31T23:27:23.109 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2023-10-31T23:27:23.134 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2023-10-31T23:27:23.201 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 979 kB/s | 90 kB 00:00 2023-10-31T23:27:23.235 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2023-10-31T23:27:23.302 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-10-31T23:27:23.385 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 241 kB/s | 20 kB 00:00 2023-10-31T23:27:23.411 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2023-10-31T23:27:26.769 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): ceph-osd-18.0.0-6795.gd1dcabba.el9.x86 20 MB/s | 117 MB 00:05 2023-10-31T23:27:26.773 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:27:26.773 INFO:teuthology.orchestra.run.smithi050.stdout:Total 21 MB/s | 133 MB 00:06 2023-10-31T23:27:26.807 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:27:26.850 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:27:26.851 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:27:27.864 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:27:27.864 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:27:29.736 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:27:29.788 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-31T23:27:29.821 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-31T23:27:29.854 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-31T23:27:29.949 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-31T23:27:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-31T23:27:30.190 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-31T23:27:30.236 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-31T23:27:30.324 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-31T23:27:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-31T23:27:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-31T23:27:30.490 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-31T23:27:30.553 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-31T23:27:30.752 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-31T23:27:30.802 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-31T23:27:30.851 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-31T23:27:30.891 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-31T23:27:30.921 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-31T23:27:31.000 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-31T23:27:31.031 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-31T23:27:31.065 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-31T23:27:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-31T23:27:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-31T23:27:31.233 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-31T23:27:31.290 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-31T23:27:31.371 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-31T23:27:31.512 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-31T23:27:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-31T23:27:31.811 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6795.gd1dcabba.el9. 28/33 2023-10-31T23:27:31.843 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/33 2023-10-31T23:27:31.868 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/33 2023-10-31T23:27:31.869 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:27:31.869 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-31T23:27:31.869 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-31T23:27:31.870 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-31T23:27:31.870 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:51.445 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/33 2023-10-31T23:27:51.477 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/33 2023-10-31T23:27:51.477 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:27:51.477 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-31T23:27:51.477 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-31T23:27:51.478 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-31T23:27:51.478 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 31/33 2023-10-31T23:27:52.774 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 31/33 2023-10-31T23:27:52.774 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:27:52.775 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-31T23:27:52.775 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-31T23:27:52.775 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-31T23:27:52.775 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:53.305 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 32/33 2023-10-31T23:27:53.333 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 32/33 2023-10-31T23:27:53.333 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:27:53.333 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-31T23:27:53.334 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-31T23:27:53.334 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-31T23:27:53.334 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:27:53.574 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 33/33 2023-10-31T23:27:59.733 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 33/33 2023-10-31T23:27:59.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/33 2023-10-31T23:27:59.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/33 2023-10-31T23:27:59.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/33 2023-10-31T23:27:59.734 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/33 2023-10-31T23:27:59.734 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 5/33 2023-10-31T23:27:59.734 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6795.gd1dcabba.el9. 6/33 2023-10-31T23:27:59.734 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-31T23:27:59.734 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-31T23:27:59.734 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-31T23:27:59.735 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-31T23:27:59.735 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-31T23:27:59.735 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-31T23:27:59.735 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-31T23:27:59.735 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-31T23:27:59.736 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-31T23:27:59.736 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-31T23:27:59.736 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-31T23:27:59.736 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-31T23:27:59.736 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-31T23:27:59.737 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-31T23:27:59.737 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-31T23:27:59.737 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-31T23:27:59.737 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-31T23:27:59.737 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-31T23:27:59.738 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-31T23:27:59.738 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-31T23:27:59.738 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-31T23:27:59.738 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-31T23:27:59.738 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-31T23:27:59.739 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-31T23:27:59.739 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-31T23:27:59.739 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-31T23:28:00.569 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-31T23:28:00.570 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:00.570 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:00.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:00.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:00.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-31T23:28:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-31T23:28:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-31T23:28:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-31T23:28:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-31T23:28:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-31T23:28:00.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-31T23:28:00.573 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-31T23:28:00.573 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-31T23:28:00.573 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-31T23:28:00.573 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-31T23:28:00.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-31T23:28:00.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-31T23:28:00.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-31T23:28:00.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-31T23:28:00.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2023-10-31T23:28:00.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-31T23:28:00.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-31T23:28:00.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-31T23:28:00.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-31T23:28:00.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-31T23:28:00.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-31T23:28:00.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-31T23:28:00.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-31T23:28:00.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-31T23:28:00.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-31T23:28:00.577 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-31T23:28:00.577 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:00.577 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:00.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2023-10-31T23:28:01.371 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:51 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:01.467 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:01.519 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:01.520 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:01.520 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:01.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2023-10-31T23:28:02.147 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:52 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:02.293 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:02.294 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:02.294 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:02.295 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:02.295 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 270 k 2023-10-31T23:28:02.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:02.295 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:02.295 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:02.296 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:02.296 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:02.296 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 270 k 2023-10-31T23:28:02.296 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 265 k 2023-10-31T23:28:02.296 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-18.0.0-6795.gd1dcabba.el9.noarch.rpm 1.1 MB/s | 270 kB 00:00 2023-10-31T23:28:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.1 MB/s | 270 kB 00:00 2023-10-31T23:28:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:02.548 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:02.548 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:02.551 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:02.552 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:02.636 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:03.050 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:03.132 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:03.527 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:03.834 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:03.834 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:03.835 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:03.835 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:03.835 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:03.835 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:04.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2023-10-31T23:28:04.670 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:54 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:04.818 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:04.819 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:04.819 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-31T23:28:04.820 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:04.820 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:04.820 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 144 k 2023-10-31T23:28:04.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:04.820 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:04.821 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:04.821 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:04.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:04.821 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 144 k 2023-10-31T23:28:04.821 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 437 k 2023-10-31T23:28:04.821 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:05.052 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-18.0.0-6795.gd1dcab 623 kB/s | 144 kB 00:00 2023-10-31T23:28:05.053 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:05.053 INFO:teuthology.orchestra.run.smithi050.stdout:Total 619 kB/s | 144 kB 00:00 2023-10-31T23:28:05.053 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:05.060 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:05.061 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:05.116 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:05.117 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:05.273 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:05.332 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba. 1/1 2023-10-31T23:28:05.357 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba. 1/1 2023-10-31T23:28:05.357 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:28:05.357 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-31T23:28:05.357 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba. 1/1 2023-10-31T23:28:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:06.308 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:06.309 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:06.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2023-10-31T23:28:07.154 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:57 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:07.249 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:07.302 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:07.303 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:07.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2023-10-31T23:28:07.935 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:57 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:08.081 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:08.082 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:08.082 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:08.082 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:08.083 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:08.083 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 1.8 M 2023-10-31T23:28:08.083 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:28:08.083 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 23 k 2023-10-31T23:28:08.083 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 15 k 2023-10-31T23:28:08.084 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-31T23:28:08.084 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-31T23:28:08.084 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-31T23:28:08.084 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2023-10-31T23:28:08.084 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-31T23:28:08.085 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:08.085 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:08.085 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:08.085 INFO:teuthology.orchestra.run.smithi050.stdout:Install 7 Packages 2023-10-31T23:28:08.085 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:08.086 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.1 M 2023-10-31T23:28:08.087 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 8.6 M 2023-10-31T23:28:08.087 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:08.652 INFO:teuthology.orchestra.run.smithi050.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6795.gd1d 113 kB/s | 23 kB 00:00 2023-10-31T23:28:08.677 INFO:teuthology.orchestra.run.smithi050.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6795.gd1dc 65 kB/s | 15 kB 00:00 2023-10-31T23:28:08.810 INFO:teuthology.orchestra.run.smithi050.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 57 kB/s | 9.0 kB 00:00 2023-10-31T23:28:08.886 INFO:teuthology.orchestra.run.smithi050.stdout:(4/7): ceph-mgr-dashboard-18.0.0-6795.gd1dcabba 4.0 MB/s | 1.8 MB 00:00 2023-10-31T23:28:08.911 INFO:teuthology.orchestra.run.smithi050.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 175 kB/s | 41 kB 00:00 2023-10-31T23:28:08.961 INFO:teuthology.orchestra.run.smithi050.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 204 kB/s | 31 kB 00:00 2023-10-31T23:28:09.070 INFO:teuthology.orchestra.run.smithi050.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2023-10-31T23:28:09.071 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:09.072 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.1 MB/s | 2.1 MB 00:00 2023-10-31T23:28:09.072 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:09.081 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:09.081 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:09.155 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:09.156 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:09.461 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:09.538 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-31T23:28:09.572 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-31T23:28:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-31T23:28:09.667 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-31T23:28:09.696 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6795.gd1dcabba.el9.n 5/7 2023-10-31T23:28:09.944 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6795.gd1dcabba.el9. 6/7 2023-10-31T23:28:09.974 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarc 7/7 2023-10-31T23:28:10.547 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarc 7/7 2023-10-31T23:28:10.547 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6795.gd1dcabba.el9. 1/7 2023-10-31T23:28:10.547 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarc 2/7 2023-10-31T23:28:10.548 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6795.gd1dcabba.el9.n 3/7 2023-10-31T23:28:10.548 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-31T23:28:10.548 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-31T23:28:10.548 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-31T23:28:10.874 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-31T23:28:10.875 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:10.875 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:10.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:10.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:10.876 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:10.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-31T23:28:10.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-31T23:28:10.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-31T23:28:10.877 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-31T23:28:10.877 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:10.877 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:11.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-31T23:28:11.684 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:01 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:11.845 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:11.846 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-31T23:28:11.846 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:11.847 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-31T23:28:11.847 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:11.847 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 7.4 M 2023-10-31T23:28:11.847 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:28:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-31T23:28:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-31T23:28:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-31T23:28:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-31T23:28:11.848 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-31T23:28:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-31T23:28:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-31T23:28:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-31T23:28:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-31T23:28:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-31T23:28:11.849 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-31T23:28:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:11.850 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:11.850 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-10-31T23:28:11.850 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2023-10-31T23:28:11.850 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:11.851 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 37 M 2023-10-31T23:28:11.851 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 209 M 2023-10-31T23:28:11.852 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:11.976 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2023-10-31T23:28:12.010 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2023-10-31T23:28:12.044 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-31T23:28:12.102 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-31T23:28:12.219 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-10-31T23:28:12.245 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 205 kB 00:00 2023-10-31T23:28:12.320 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-10-31T23:28:12.345 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-31T23:28:12.404 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-10-31T23:28:12.571 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2023-10-31T23:28:12.604 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 987 kB/s | 195 kB 00:00 2023-10-31T23:28:12.771 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 8.0 MB/s | 7.4 MB 00:00 2023-10-31T23:28:12.774 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:12.774 INFO:teuthology.orchestra.run.smithi050.stdout:Total 40 MB/s | 37 MB 00:00 2023-10-31T23:28:12.775 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:12.799 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:12.799 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:13.239 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:13.239 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:13.960 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:14.028 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-31T23:28:14.067 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-31T23:28:14.167 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-31T23:28:14.204 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-31T23:28:14.394 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-31T23:28:14.428 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-31T23:28:14.525 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-31T23:28:15.210 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-31T23:28:15.320 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-31T23:28:16.624 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-31T23:28:17.702 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-31T23:28:17.736 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dca 12/12 2023-10-31T23:28:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dca 12/12 2023-10-31T23:28:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dca 1/12 2023-10-31T23:28:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-31T23:28:20.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-31T23:28:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-31T23:28:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-31T23:28:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-31T23:28:20.446 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-31T23:28:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-31T23:28:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-31T23:28:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-31T23:28:20.447 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-31T23:28:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-31T23:28:20.909 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:20.909 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-31T23:28:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-31T23:28:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-31T23:28:20.910 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-31T23:28:20.911 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-31T23:28:20.911 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-31T23:28:20.911 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-31T23:28:20.911 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-31T23:28:20.911 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-31T23:28:20.911 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-31T23:28:20.912 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-31T23:28:20.912 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:20.912 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:21.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2023-10-31T23:28:21.857 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:11 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:22.014 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:22.016 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-31T23:28:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:22.016 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-31T23:28:22.016 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:22.016 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 48 k 2023-10-31T23:28:22.017 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:28:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-31T23:28:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-31T23:28:22.017 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-31T23:28:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-31T23:28:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-31T23:28:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-31T23:28:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-31T23:28:22.018 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-31T23:28:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-31T23:28:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-31T23:28:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-31T23:28:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-31T23:28:22.019 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:22.019 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:22.020 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-31T23:28:22.020 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2023-10-31T23:28:22.020 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:22.021 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.0 M 2023-10-31T23:28:22.021 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2023-10-31T23:28:22.021 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:22.321 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2023-10-31T23:28:22.337 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 125 kB/s | 18 kB 00:00 2023-10-31T23:28:22.379 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2023-10-31T23:28:22.405 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-10-31T23:28:22.430 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-31T23:28:22.447 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2023-10-31T23:28:22.464 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): ceph-mgr-rook-18.0.0-6795.gd1dcabba.el9 176 kB/s | 48 kB 00:00 2023-10-31T23:28:22.556 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2023-10-31T23:28:22.573 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 226 kB/s | 32 kB 00:00 2023-10-31T23:28:22.665 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 645 kB/s | 59 kB 00:00 2023-10-31T23:28:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 872 kB/s | 197 kB 00:00 2023-10-31T23:28:22.757 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 975 kB/s | 90 kB 00:00 2023-10-31T23:28:22.824 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2023-10-31T23:28:22.826 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:22.826 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2023-10-31T23:28:22.827 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:22.840 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:22.840 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:22.946 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:22.947 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:23.475 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:23.653 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-31T23:28:23.711 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-31T23:28:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-31T23:28:23.801 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-31T23:28:23.843 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-31T23:28:23.939 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-31T23:28:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-31T23:28:24.111 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-31T23:28:24.870 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-31T23:28:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-31T23:28:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-31T23:28:25.005 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-31T23:28:25.037 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 13/13 2023-10-31T23:28:25.759 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 13/13 2023-10-31T23:28:25.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 1/13 2023-10-31T23:28:25.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-31T23:28:25.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-31T23:28:25.760 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-31T23:28:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-31T23:28:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-31T23:28:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-31T23:28:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-31T23:28:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-31T23:28:25.761 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-31T23:28:25.762 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-31T23:28:25.762 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-31T23:28:26.140 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-31T23:28:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:26.141 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:26.141 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-31T23:28:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-31T23:28:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-31T23:28:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-31T23:28:26.142 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-31T23:28:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-31T23:28:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-31T23:28:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-31T23:28:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-31T23:28:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-31T23:28:26.143 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-31T23:28:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-31T23:28:26.144 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:26.144 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:26.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2023-10-31T23:28:26.987 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:16 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:27.136 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:27.137 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-31T23:28:27.137 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:27.137 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-31T23:28:27.138 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:27.138 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 132 k 2023-10-31T23:28:27.138 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:28:27.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-31T23:28:27.139 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-31T23:28:27.139 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-31T23:28:27.139 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-31T23:28:27.139 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-31T23:28:27.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:27.139 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:27.140 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2023-10-31T23:28:27.140 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-10-31T23:28:27.140 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:27.141 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 6.8 M 2023-10-31T23:28:27.141 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2023-10-31T23:28:27.141 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:27.335 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.6 MB/s | 228 kB 00:00 2023-10-31T23:28:27.443 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2023-10-31T23:28:27.527 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6795.gd1dcabba.e 396 kB/s | 132 kB 00:00 2023-10-31T23:28:27.785 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2023-10-31T23:28:27.860 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2023-10-31T23:28:27.911 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 952 kB/s | 548 kB 00:00 2023-10-31T23:28:27.913 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:27.914 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.7 MB/s | 6.8 MB 00:00 2023-10-31T23:28:27.914 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:27.924 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:27.925 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:28.024 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:28.026 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:28.335 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:28.516 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-31T23:28:28.563 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-31T23:28:29.110 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-31T23:28:29.209 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-31T23:28:29.277 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-31T23:28:29.300 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 6/6 2023-10-31T23:28:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 6/6 2023-10-31T23:28:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/6 2023-10-31T23:28:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-31T23:28:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-31T23:28:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-31T23:28:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-31T23:28:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-31T23:28:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:30.430 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-31T23:28:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-31T23:28:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-31T23:28:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-31T23:28:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-31T23:28:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:30.432 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:30.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2023-10-31T23:28:31.251 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:21 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:31.402 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:31.403 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:31.403 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:31.403 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:31.403 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 832 k 2023-10-31T23:28:31.404 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:28:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-31T23:28:31.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:31.404 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:31.405 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:31.405 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2023-10-31T23:28:31.405 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:31.405 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 910 k 2023-10-31T23:28:31.405 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 3.0 M 2023-10-31T23:28:31.406 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:31.529 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 628 kB/s | 79 kB 00:00 2023-10-31T23:28:31.754 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-18.0.0-6795.gd1dcabba.el9.x86_ 2.3 MB/s | 832 kB 00:00 2023-10-31T23:28:31.755 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:31.756 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.5 MB/s | 910 kB 00:00 2023-10-31T23:28:31.756 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:31.778 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:31.779 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:31.859 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:31.859 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:32.046 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:32.184 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-31T23:28:32.257 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/2 2023-10-31T23:28:32.986 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/2 2023-10-31T23:28:32.987 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/2 2023-10-31T23:28:33.321 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-31T23:28:33.321 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:33.321 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:33.321 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-31T23:28:33.321 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:33.322 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:33.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2023-10-31T23:28:34.185 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:24 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:34.331 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:34.332 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:34.332 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:28:34.333 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:34.333 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:34.333 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 262 k 2023-10-31T23:28:34.333 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:34.333 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:34.334 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:34.334 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:34.334 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:34.334 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 262 k 2023-10-31T23:28:34.334 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.2 M 2023-10-31T23:28:34.335 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:34.584 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-18.0.0-6795.gd1dcabba.el9.noarch.rp 1.0 MB/s | 262 kB 00:00 2023-10-31T23:28:34.585 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:34.585 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-10-31T23:28:34.585 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:34.593 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:34.594 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:34.627 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:34.628 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:34.851 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:34.927 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:34.944 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:28:34.944 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-31T23:28:34.944 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-10-31T23:28:36.070 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-10-31T23:28:36.071 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:36.071 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:36.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2023-10-31T23:28:36.969 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:26 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:37.116 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:37.117 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-31T23:28:37.117 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:37.117 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 127 k 2023-10-31T23:28:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:37.118 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:37.118 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:37.118 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:37.119 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 127 k 2023-10-31T23:28:37.119 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2023-10-31T23:28:37.119 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:37.322 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-18.0.0-6795.gd1dcabba.el9.x86_64 618 kB/s | 127 kB 00:00 2023-10-31T23:28:37.323 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:37.324 INFO:teuthology.orchestra.run.smithi050.stdout:Total 613 kB/s | 127 kB 00:00 2023-10-31T23:28:37.324 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:37.329 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:37.330 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:37.388 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:37.388 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:37.542 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:38.089 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:38.404 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:38.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:38.404 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:38.404 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:38.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:38.405 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:38.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2023-10-31T23:28:39.260 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:29 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:39.354 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:39.406 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:39.407 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:39.407 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:39.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2023-10-31T23:28:40.060 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:30 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:40.207 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:40.208 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:40.208 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-31T23:28:40.208 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:40.209 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:40.209 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 31 k 2023-10-31T23:28:40.209 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:40.209 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:40.209 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:40.210 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:40.210 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:40.210 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 k 2023-10-31T23:28:40.210 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 139 k 2023-10-31T23:28:40.210 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:40.385 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-18.0.0-6795.gd1dcabba.el9.x86_6 178 kB/s | 31 kB 00:00 2023-10-31T23:28:40.385 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:40.386 INFO:teuthology.orchestra.run.smithi050.stdout:Total 176 kB/s | 31 kB 00:00 2023-10-31T23:28:40.386 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:40.389 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:40.389 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:40.427 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:40.427 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:40.513 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:40.643 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:41.009 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:41.294 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:41.295 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:41.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:41.295 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:41.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2023-10-31T23:28:42.068 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:32 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:42.163 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:42.216 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:42.217 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:42.217 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:42.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2023-10-31T23:28:42.862 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:32 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:42.958 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:43.011 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:43.012 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:43.012 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:43.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2023-10-31T23:28:43.673 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:33 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:43.767 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:43.819 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:43.820 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:43.820 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:43.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2023-10-31T23:28:44.479 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:34 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:44.573 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:44.625 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:44.627 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:44.627 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:44.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2023-10-31T23:28:45.284 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:35 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:45.378 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:45.431 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:45.432 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:45.432 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:45.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2023-10-31T23:28:46.098 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:36 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:46.192 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-10-31T23:28:46.245 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:46.246 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-10-31T23:28:46.246 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:46.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2023-10-31T23:28:46.904 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:36 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:47.052 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:47.053 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:47.053 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-31T23:28:47.053 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:47.053 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:47.054 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 87 k 2023-10-31T23:28:47.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:47.054 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:47.054 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:47.054 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:47.055 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:47.055 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 87 k 2023-10-31T23:28:47.055 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 252 k 2023-10-31T23:28:47.055 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:47.304 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-18.0.0-6795.gd1dcabba.el9.x86_64.rpm 346 kB/s | 87 kB 00:00 2023-10-31T23:28:47.305 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:47.305 INFO:teuthology.orchestra.run.smithi050.stdout:Total 344 kB/s | 87 kB 00:00 2023-10-31T23:28:47.306 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:47.313 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:47.313 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:47.371 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:47.371 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:47.510 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:47.597 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:47.983 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:48.318 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:48.318 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:48.318 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:48.318 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:48.319 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:48.319 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:48.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2023-10-31T23:28:49.133 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:39 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:49.279 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:49.280 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:49.280 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-10-31T23:28:49.281 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:49.281 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:49.281 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 3.0 M 2023-10-31T23:28:49.281 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:49.281 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:49.282 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:49.282 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:49.282 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:49.282 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.0 M 2023-10-31T23:28:49.282 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 13 M 2023-10-31T23:28:49.283 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:50.007 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-18.0.0-6795.gd1dcabba.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2023-10-31T23:28:50.009 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:50.009 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.1 MB/s | 3.0 MB 00:00 2023-10-31T23:28:50.010 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:50.018 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:50.018 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:50.100 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:50.101 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:50.554 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:50.633 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:50.658 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:50.658 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T23:28:50.658 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-31T23:28:50.658 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-31T23:28:50.659 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-31T23:28:50.659 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:51.799 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:51.799 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:51.799 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:51.800 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:51.800 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:51.800 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:52.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2023-10-31T23:28:52.676 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:42 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:28:52.829 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:28:52.830 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:52.830 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-10-31T23:28:52.830 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:52.831 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:28:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 168 k 2023-10-31T23:28:52.831 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:52.831 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:28:52.832 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:28:52.832 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-10-31T23:28:52.832 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:52.832 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 168 k 2023-10-31T23:28:52.832 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 532 k 2023-10-31T23:28:52.833 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:28:53.078 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-18.0.0-6795.gd1dcabba.el9.x86_64.rpm 682 kB/s | 168 kB 00:00 2023-10-31T23:28:53.079 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:28:53.080 INFO:teuthology.orchestra.run.smithi050.stdout:Total 678 kB/s | 168 kB 00:00 2023-10-31T23:28:53.080 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:28:53.088 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:28:53.088 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:28:53.147 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:28:53.147 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:28:53.318 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:28:53.429 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:54.201 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-10-31T23:28:54.202 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:54.202 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:28:54.202 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:28:54.202 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:54.203 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:28:54.439 DEBUG:teuthology.parallel:result is None 2023-10-31T23:28:54.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 2023-10-31T23:28:54.588 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-31T23:28:54.610 INFO:teuthology.orchestra.run.smithi050.stdout:18.0.0-6795.gd1dcabba.el9 2023-10-31T23:28:54.610 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6795.gd1dcabba.el9 2023-10-31T23:28:54.611 INFO:teuthology.task.install:The correct ceph version 18.0.0-6795.gd1dcabba is installed. 2023-10-31T23:28:54.611 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-31T23:28:54.612 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:54.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-31T23:28:54.682 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-31T23:28:54.683 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:54.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2023-10-31T23:28:54.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-31T23:28:54.823 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-31T23:28:54.824 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:54.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-31T23:28:54.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-31T23:28:54.964 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-31T23:28:54.965 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:54.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2023-10-31T23:28:55.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-31T23:28:55.106 INFO:teuthology.run_tasks:Running task ceph... 2023-10-31T23:28:55.179 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-31T23:28:55.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2023-10-31T23:28:55.210 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-31T23:28:55.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-31T23:28:55.280 INFO:tasks.ceph:Creating extra log directories... 2023-10-31T23:28:55.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-31T23:28:55.358 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-31T23:28:55.358 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-31T23:28:55.359 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530', 'branch': 'wip-matanb-crimson-do_osd_ops_execute-part3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw}', 'email': None, 'first_in_suite': False, 'job_id': '7442530', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-do_osd_ops_execute-part3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'a9edd23b5a8ddacab225824948b06b370baf276b'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6029, 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'a9edd23b5a8ddacab225824948b06b370baf276b', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsgaC1yCuZUf5QYeSnmN1ltMdYBBt5S9+cR1tpCFj9yPqIVSYa6vAS/p0ZO964kV4eGYOEkhTu+Nl4PFeTXpx0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-31_22:58:07', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-31T23:28:55.360 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-31T23:28:55.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2023-10-31T23:28:55.477 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:55.477 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2023-10-31T23:28:55.533 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-31T23:28:55.533 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2023-10-31T23:28:55.588 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-31T23:28:55.589 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T23:28:55.589 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-10-31T23:28:55.589 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T23:28:55.589 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T23:28:55.589 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-31 23:28:50.904244421 +0000 2023-10-31T23:28:55.590 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-31 23:25:19.912075492 +0000 2023-10-31T23:28:55.590 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-31 23:25:19.912075492 +0000 2023-10-31T23:28:55.590 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-31 23:25:19.912075492 +0000 2023-10-31T23:28:55.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-31T23:28:55.657 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-31T23:28:55.657 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-31T23:28:55.657 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.3883e-05 s, 5.5 MB/s 2023-10-31T23:28:55.658 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-31T23:28:55.714 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2023-10-31T23:28:55.769 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-31T23:28:55.769 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T23:28:55.770 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-10-31T23:28:55.770 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T23:28:55.770 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T23:28:55.770 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-31 23:28:50.904244421 +0000 2023-10-31T23:28:55.770 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-31 23:25:20.410062752 +0000 2023-10-31T23:28:55.771 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-31 23:25:20.410062752 +0000 2023-10-31T23:28:55.771 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-31 23:25:20.410062752 +0000 2023-10-31T23:28:55.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-31T23:28:55.838 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-31T23:28:55.838 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-31T23:28:55.839 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000102018 s, 5.0 MB/s 2023-10-31T23:28:55.840 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-31T23:28:55.896 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2023-10-31T23:28:55.951 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-31T23:28:55.952 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T23:28:55.952 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-10-31T23:28:55.952 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T23:28:55.952 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T23:28:55.952 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-31 23:28:50.904244421 +0000 2023-10-31T23:28:55.953 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-31 23:25:20.912049911 +0000 2023-10-31T23:28:55.953 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-31 23:25:20.912049911 +0000 2023-10-31T23:28:55.953 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-31 23:25:20.912049911 +0000 2023-10-31T23:28:55.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-31T23:28:56.020 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-31T23:28:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-31T23:28:56.021 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 7.6995e-05 s, 6.6 MB/s 2023-10-31T23:28:56.022 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-31T23:28:56.078 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2023-10-31T23:28:56.133 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-31T23:28:56.133 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T23:28:56.134 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-10-31T23:28:56.134 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T23:28:56.134 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T23:28:56.134 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-10-31 23:28:50.905244396 +0000 2023-10-31T23:28:56.135 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-10-31 23:25:21.387037760 +0000 2023-10-31T23:28:56.135 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-10-31 23:25:21.387037760 +0000 2023-10-31T23:28:56.135 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2023-10-31 23:25:21.387037760 +0000 2023-10-31T23:28:56.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-31T23:28:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-10-31T23:28:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-10-31T23:28:56.203 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.2164e-05 s, 6.2 MB/s 2023-10-31T23:28:56.204 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-31T23:28:56.260 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T23:28:56.260 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-31T23:28:56.261 INFO:tasks.ceph:Generating config... 2023-10-31T23:28:56.262 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-31T23:28:56.263 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-31T23:28:56.263 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-10-31T23:28:56.263 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-31T23:28:56.263 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-31T23:28:56.264 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-10-31T23:28:56.264 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-31T23:28:56.264 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-31T23:28:56.264 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-31T23:28:56.265 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-10-31T23:28:56.265 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-10-31T23:28:56.265 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-10-31T23:28:56.265 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-10-31T23:28:56.266 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-31T23:28:56.266 INFO:tasks.ceph:[osd] err to stderr = True 2023-10-31T23:28:56.266 INFO:tasks.ceph:[osd] log flush on exit = True 2023-10-31T23:28:56.266 INFO:tasks.ceph:[osd] log to file = False 2023-10-31T23:28:56.267 INFO:tasks.ceph:[osd] log to stderr = True 2023-10-31T23:28:56.267 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-31T23:28:56.267 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-31T23:28:56.267 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-31T23:28:56.268 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-31T23:28:56.268 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-31T23:28:56.268 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-31T23:28:56.268 INFO:tasks.ceph:[global] auth client required = none 2023-10-31T23:28:56.269 INFO:tasks.ceph:[global] auth cluster required = none 2023-10-31T23:28:56.269 INFO:tasks.ceph:[global] auth service required = none 2023-10-31T23:28:56.269 INFO:tasks.ceph:[global] auth supported = none 2023-10-31T23:28:56.269 INFO:tasks.ceph:[global] debug asok = 0/0 2023-10-31T23:28:56.270 INFO:tasks.ceph:[global] debug auth = 0/0 2023-10-31T23:28:56.270 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-10-31T23:28:56.270 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-10-31T23:28:56.271 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-10-31T23:28:56.271 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-10-31T23:28:56.271 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-10-31T23:28:56.271 INFO:tasks.ceph:[global] debug client = 0/0 2023-10-31T23:28:56.272 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-10-31T23:28:56.272 INFO:tasks.ceph:[global] debug context = 0/0 2023-10-31T23:28:56.272 INFO:tasks.ceph:[global] debug crush = 0/0 2023-10-31T23:28:56.272 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-10-31T23:28:56.273 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-10-31T23:28:56.273 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-10-31T23:28:56.273 INFO:tasks.ceph:[global] debug filer = 0/0 2023-10-31T23:28:56.273 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-10-31T23:28:56.274 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-10-31T23:28:56.274 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-10-31T23:28:56.274 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-10-31T23:28:56.274 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-10-31T23:28:56.275 INFO:tasks.ceph:[global] debug journal = 0/0 2023-10-31T23:28:56.275 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-10-31T23:28:56.275 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-10-31T23:28:56.276 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-10-31T23:28:56.276 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-10-31T23:28:56.276 INFO:tasks.ceph:[global] debug mds = 0/0 2023-10-31T23:28:56.276 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-10-31T23:28:56.277 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-10-31T23:28:56.277 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-10-31T23:28:56.277 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-10-31T23:28:56.278 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-10-31T23:28:56.278 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-10-31T23:28:56.279 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-10-31T23:28:56.279 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-10-31T23:28:56.279 INFO:tasks.ceph:[global] debug mon = 0/0 2023-10-31T23:28:56.279 INFO:tasks.ceph:[global] debug monc = 0/0 2023-10-31T23:28:56.280 INFO:tasks.ceph:[global] debug ms = 0/0 2023-10-31T23:28:56.280 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-10-31T23:28:56.280 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-10-31T23:28:56.281 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-10-31T23:28:56.281 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-10-31T23:28:56.281 INFO:tasks.ceph:[global] debug osd = 0/0 2023-10-31T23:28:56.281 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-10-31T23:28:56.282 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-10-31T23:28:56.282 INFO:tasks.ceph:[global] debug rados = 0/0 2023-10-31T23:28:56.282 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-10-31T23:28:56.282 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-10-31T23:28:56.283 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-10-31T23:28:56.283 INFO:tasks.ceph:[global] debug refs = 0/0 2023-10-31T23:28:56.283 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-10-31T23:28:56.284 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-10-31T23:28:56.284 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-10-31T23:28:56.284 INFO:tasks.ceph:[global] debug striper = 0/0 2023-10-31T23:28:56.284 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-10-31T23:28:56.285 INFO:tasks.ceph:[global] debug timer = 0/0 2023-10-31T23:28:56.285 INFO:tasks.ceph:[global] debug tp = 0/0 2023-10-31T23:28:56.285 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-31T23:28:56.285 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-31T23:28:56.286 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-31T23:28:56.286 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-31T23:28:56.286 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-31T23:28:56.286 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-31T23:28:56.287 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-31T23:28:56.287 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-31T23:28:56.287 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-31T23:28:56.287 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-10-31T23:28:56.288 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-10-31T23:28:56.288 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-10-31T23:28:56.288 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-31T23:28:56.289 INFO:tasks.ceph:Setting up mon.a... 2023-10-31T23:28:56.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-31T23:28:56.361 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2023-10-31T23:28:56.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-31T23:28:56.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-31T23:28:56.448 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.50')] 2023-10-31T23:28:56.449 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-31T23:28:56.450 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:56.450 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-31T23:28:56.503 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:28:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:28:56.587+0000 7feb48e37e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:28:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:28:56.588+0000 7feb48e37e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:28:56.589 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:28:56.588+0000 7feb48e37e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:28:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:28:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a 2023-10-31T23:28:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = pacific 2023-10-31T23:28:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2023-10-31T23:28:56.592 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a 2023-10-31T23:28:56.592 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2023-10-31T23:28:56.588506+0000 2023-10-31T23:28:56.592 INFO:teuthology.orchestra.run.smithi050.stdout:created 2023-10-31T23:28:56.588506+0000 2023-10-31T23:28:56.592 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 16 (pacific) 2023-10-31T23:28:56.593 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2023-10-31T23:28:56.593 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2023-10-31T23:28:56.593 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-31T23:28:56.594 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-31T23:28:56.609 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a... 2023-10-31T23:28:56.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-31T23:28:56.709 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-10-31T23:28:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-10-31T23:28:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-31T23:28:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = none 2023-10-31T23:28:56.710 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-10-31T23:28:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-10-31T23:28:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-10-31T23:28:56.711 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-10-31T23:28:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-10-31T23:28:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-10-31T23:28:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-31T23:28:56.712 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-10-31T23:28:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-10-31T23:28:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-31T23:28:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-10-31T23:28:56.713 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-10-31T23:28:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-10-31T23:28:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-10-31T23:28:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-10-31T23:28:56.714 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-10-31T23:28:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-10-31T23:28:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-10-31T23:28:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-31T23:28:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-31T23:28:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-10-31T23:28:56.715 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-31T23:28:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-10-31T23:28:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-10-31T23:28:56.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-10-31T23:28:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-10-31T23:28:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-10-31T23:28:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = 172.21.15.50 2023-10-31T23:28:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: auth client required = none 2023-10-31T23:28:56.717 INFO:teuthology.orchestra.run.smithi050.stdout: auth cluster required = none 2023-10-31T23:28:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: auth service required = none 2023-10-31T23:28:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: debug asok = 0/0 2023-10-31T23:28:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 0/0 2023-10-31T23:28:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: debug bdev = 0/0 2023-10-31T23:28:56.718 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 0/0 2023-10-31T23:28:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 0/0 2023-10-31T23:28:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: debug buffer = 0/0 2023-10-31T23:28:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: debug civetweb = 0/0 2023-10-31T23:28:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: debug client = 0/0 2023-10-31T23:28:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: debug compressor = 0/0 2023-10-31T23:28:56.719 INFO:teuthology.orchestra.run.smithi050.stdout: debug context = 0/0 2023-10-31T23:28:56.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug crush = 0/0 2023-10-31T23:28:56.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug crypto = 0/0 2023-10-31T23:28:56.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug dpdk = 0/0 2023-10-31T23:28:56.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug eventtrace = 0/0 2023-10-31T23:28:56.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug filer = 0/0 2023-10-31T23:28:56.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug filestore = 0/0 2023-10-31T23:28:56.721 INFO:teuthology.orchestra.run.smithi050.stdout: debug finisher = 0/0 2023-10-31T23:28:56.721 INFO:teuthology.orchestra.run.smithi050.stdout: debug fuse = 0/0 2023-10-31T23:28:56.721 INFO:teuthology.orchestra.run.smithi050.stdout: debug heartbeatmap = 0/0 2023-10-31T23:28:56.721 INFO:teuthology.orchestra.run.smithi050.stdout: debug javaclient = 0/0 2023-10-31T23:28:56.721 INFO:teuthology.orchestra.run.smithi050.stdout: debug journal = 0/0 2023-10-31T23:28:56.721 INFO:teuthology.orchestra.run.smithi050.stdout: debug journaler = 0/0 2023-10-31T23:28:56.722 INFO:teuthology.orchestra.run.smithi050.stdout: debug kstore = 0/0 2023-10-31T23:28:56.722 INFO:teuthology.orchestra.run.smithi050.stdout: debug leveldb = 0/0 2023-10-31T23:28:56.722 INFO:teuthology.orchestra.run.smithi050.stdout: debug lockdep = 0/0 2023-10-31T23:28:56.722 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds = 0/0 2023-10-31T23:28:56.722 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds balancer = 0/0 2023-10-31T23:28:56.723 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds locker = 0/0 2023-10-31T23:28:56.723 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds log = 0/0 2023-10-31T23:28:56.723 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds log expire = 0/0 2023-10-31T23:28:56.723 INFO:teuthology.orchestra.run.smithi050.stdout: debug mds migrator = 0/0 2023-10-31T23:28:56.723 INFO:teuthology.orchestra.run.smithi050.stdout: debug memdb = 0/0 2023-10-31T23:28:56.723 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 0/0 2023-10-31T23:28:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgrc = 0/0 2023-10-31T23:28:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 0/0 2023-10-31T23:28:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 0/0 2023-10-31T23:28:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 0/0 2023-10-31T23:28:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug objclass = 0/0 2023-10-31T23:28:56.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug objectcacher = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug objecter = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug optracker = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug perfcounter = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug rados = 0/0 2023-10-31T23:28:56.725 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd = 0/0 2023-10-31T23:28:56.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd mirror = 0/0 2023-10-31T23:28:56.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug rbd replay = 0/0 2023-10-31T23:28:56.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug refs = 0/0 2023-10-31T23:28:56.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug rgw = 0/0 2023-10-31T23:28:56.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug rgw sync = 0/0 2023-10-31T23:28:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 0/0 2023-10-31T23:28:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug striper = 0/0 2023-10-31T23:28:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug throttle = 0/0 2023-10-31T23:28:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug timer = 0/0 2023-10-31T23:28:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug tp = 0/0 2023-10-31T23:28:56.727 INFO:teuthology.orchestra.run.smithi050.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-31T23:28:56.728 INFO:teuthology.orchestra.run.smithi050.stdout: ms client mode = crc 2023-10-31T23:28:56.728 INFO:teuthology.orchestra.run.smithi050.stdout: ms cluster mode = crc 2023-10-31T23:28:56.728 INFO:teuthology.orchestra.run.smithi050.stdout: ms mon client mode = crc 2023-10-31T23:28:56.728 INFO:teuthology.orchestra.run.smithi050.stdout: ms mon cluster mode = crc 2023-10-31T23:28:56.728 INFO:teuthology.orchestra.run.smithi050.stdout: ms mon service mode = crc 2023-10-31T23:28:56.728 INFO:teuthology.orchestra.run.smithi050.stdout: ms service mode = crc 2023-10-31T23:28:56.729 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a 2023-10-31T23:28:56.729 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.729 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-10-31T23:28:56.729 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-10-31T23:28:56.729 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.729 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-10-31T23:28:56.730 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-10-31T23:28:56.730 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-10-31T23:28:56.730 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.730 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-10-31T23:28:56.730 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-10-31T23:28:56.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.731 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-10-31T23:28:56.731 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-10-31T23:28:56.731 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-10-31T23:28:56.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.731 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-10-31T23:28:56.732 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-10-31T23:28:56.732 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.732 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-10-31T23:28:56.732 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.732 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-10-31T23:28:56.732 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-10-31T23:28:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-10-31T23:28:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-10-31T23:28:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.733 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-10-31T23:28:56.734 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-10-31T23:28:56.734 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2023-10-31T23:28:56.734 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-10-31T23:28:56.734 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression algorithm = snappy 2023-10-31T23:28:56.734 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression mode = aggressive 2023-10-31T23:28:56.735 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-10-31T23:28:56.735 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore rocksdb cf = False 2023-10-31T23:28:56.735 INFO:teuthology.orchestra.run.smithi050.stdout: crimson seastar smp = 3 2023-10-31T23:28:56.735 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2023-10-31T23:28:56.735 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2023-10-31T23:28:56.735 INFO:teuthology.orchestra.run.smithi050.stdout: debug filestore = 0/0 2023-10-31T23:28:56.736 INFO:teuthology.orchestra.run.smithi050.stdout: debug journal = 0/0 2023-10-31T23:28:56.736 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 0/0 2023-10-31T23:28:56.736 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 0/0 2023-10-31T23:28:56.736 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2023-10-31T23:28:56.736 INFO:teuthology.orchestra.run.smithi050.stdout: err to stderr = True 2023-10-31T23:28:56.736 INFO:teuthology.orchestra.run.smithi050.stdout: log flush on exit = True 2023-10-31T23:28:56.737 INFO:teuthology.orchestra.run.smithi050.stdout: log to file = False 2023-10-31T23:28:56.737 INFO:teuthology.orchestra.run.smithi050.stdout: log to stderr = True 2023-10-31T23:28:56.737 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-10-31T23:28:56.737 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-10-31T23:28:56.737 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-10-31T23:28:56.737 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-10-31T23:28:56.738 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-10-31T23:28:56.738 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2023-10-31T23:28:56.738 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.738 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-10-31T23:28:56.738 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-10-31T23:28:56.739 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-10-31T23:28:56.739 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-10-31T23:28:56.739 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-31T23:28:56.739 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-31T23:28:56.739 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-31T23:28:56.739 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-10-31T23:28:56.740 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.740 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-10-31T23:28:56.740 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 0/0 2023-10-31T23:28:56.740 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 0/0 2023-10-31T23:28:56.740 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 0/0 2023-10-31T23:28:56.740 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-10-31T23:28:56.741 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-10-31T23:28:56.741 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-10-31T23:28:56.741 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-10-31T23:28:56.741 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-10-31T23:28:56.741 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-10-31T23:28:56.741 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-10-31T23:28:56.742 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.742 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-31T23:28:56.742 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-10-31T23:28:56.742 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-10-31T23:28:56.742 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.743 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-10-31T23:28:56.743 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-31T23:28:56.743 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-31T23:28:56.743 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.743 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-10-31T23:28:56.743 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-10-31T23:28:56.744 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.744 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-10-31T23:28:56.744 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default crimson = True 2023-10-31T23:28:56.744 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:28:56.744 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-10-31T23:28:56.744 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-10-31T23:28:56.745 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-10-31T23:28:56.745 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-10-31T23:28:56.745 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-31T23:28:56.745 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-31T23:28:56.745 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-10-31T23:28:56.746 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-31T23:28:56.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-31T23:28:56.851 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-31T23:28:56.852 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:56.852 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-31T23:28:56.867 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:56.868 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-31T23:28:56.922 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:28:56.923 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:56.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-31T23:28:56.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-31T23:28:57.008 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:28:57.009 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:28:57.063 INFO:tasks.ceph:Setting up mon nodes... 2023-10-31T23:28:57.064 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-31T23:28:57.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-31T23:28:57.171 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-31T23:28:57.173 INFO:tasks.ceph:Setting up mds nodes... 2023-10-31T23:28:57.174 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-31T23:28:57.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-31T23:28:57.226 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-31T23:28:57.305 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-31T23:28:57.305 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-31T23:28:57.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-31T23:28:57.336 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T23:28:57.336 INFO:tasks.ceph:role: osd.0 2023-10-31T23:28:57.336 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:28:57.336 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-31T23:28:57.406 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T23:28:57.406 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T23:28:57.407 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T23:28:57.407 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T23:28:57.407 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T23:28:57.407 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-31T23:28:57.407 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T23:28:57.408 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T23:28:57.408 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T23:28:57.408 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T23:28:57.438 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-31T23:28:57.440 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-31T23:28:57.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-31T23:28:57.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-31T23:28:57.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-31T23:28:57.662 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T23:28:57.662 INFO:tasks.ceph:role: osd.1 2023-10-31T23:28:57.663 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:28:57.663 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-31T23:28:57.732 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T23:28:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T23:28:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T23:28:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T23:28:57.733 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T23:28:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-31T23:28:57.733 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T23:28:57.734 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T23:28:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T23:28:57.734 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T23:28:57.765 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-31T23:28:57.767 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-31T23:28:57.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-31T23:28:57.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-31T23:28:57.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-31T23:28:57.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T23:28:57.990 INFO:tasks.ceph:role: osd.2 2023-10-31T23:28:57.990 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:28:57.990 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-31T23:28:58.059 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T23:28:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T23:28:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T23:28:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T23:28:58.060 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T23:28:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-10-31T23:28:58.060 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T23:28:58.061 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T23:28:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T23:28:58.061 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T23:28:58.093 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-10-31T23:28:58.095 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-10-31T23:28:58.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-31T23:28:58.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-31T23:28:58.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:28:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T23:28:58.827 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:58,827 seastar - Reactor backend: linux-aio 2023-10-31T23:28:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:58,842 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T23:28:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:58,842 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T23:28:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:58,842 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T23:28:58.843 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:58,842 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T23:28:58.845 INFO:teuthology.orchestra.run.smithi050.stderr:==79717==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T23:28:58.851 INFO:teuthology.orchestra.run.smithi050.stderr:==79717==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffed3b6b000; bottom 0x7fbeefaf1000; size: 0x003fe407a000 (274408644608) 2023-10-31T23:28:58.852 INFO:teuthology.orchestra.run.smithi050.stderr:False positive error reports may follow 2023-10-31T23:28:58.852 INFO:teuthology.orchestra.run.smithi050.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T23:28:58.877 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x22aa87b9 0x22aa9006 0x22a84779 0x22a84e30 0x22a95a9c 0x22a97580 0x22a9adb3 0x22ae4cc4 0x22aae595 0x22ab1bbe 0x22ac067f 0x22ae0321 0x22ae0ba9 0x22a93ca9 0x22aa32c9 0x22aa49c5 0x22aa539e 0x22aa6489 0x22bbd8d8 0x22c5122d 0x229b6079 0x22a8ffff 0x22a92428 0x22a7eb74 0x22a7e1cc 0x22920b9e 0x247b7df3 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1ec5c758 0x1ec668c5 0x1eae33c9 0x3feaf 0x3ff5f 0x1e59ba04 2023-10-31T23:28:58.878 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:28:59.392 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:28:59.392 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2023-10-31T23:28:59.393 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2023-10-31T23:28:59.393 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2023-10-31T23:28:59.393 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:28:59.393 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:28:59.448 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T23:28:59.449 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:59,448 seastar - Reactor backend: linux-aio 2023-10-31T23:28:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:59,466 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T23:28:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:59,466 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T23:28:59.467 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:59,466 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T23:28:59.474 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:28:59,474 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T23:28:59.477 INFO:teuthology.orchestra.run.smithi050.stderr:==79715==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T23:28:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:==79715==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffed3b6b000; bottom 0x7fbeed8da000; size: 0x003fe6291000 (274444390400) 2023-10-31T23:28:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:False positive error reports may follow 2023-10-31T23:28:59.483 INFO:teuthology.orchestra.run.smithi050.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T23:28:59.509 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x227ba569 0x228b71f1 0x247b8898 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1eae4c7f 0x3feaf 0x3ff5f 0x1e59ba04 2023-10-31T23:28:59.509 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:28:59.543 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x3976e 0x3f249 0x226c2b01 0x22736f35 0x2725f0a5 0x272defdf 0x25e81878 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-10-31T23:28:59.543 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:28:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 90 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2604eb84 0x260527fc 0x26055843 0x25e854b7 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-10-31T23:28:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:03.547 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:03.547 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2023-10-31T23:29:03.547 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2023-10-31T23:29:03.547 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2023-10-31T23:29:03.548 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:03.548 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:03.548 INFO:teuthology.orchestra.run.smithi050.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a 2023-10-31T23:29:03.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-31T23:29:03.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:29:04.157 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T23:29:04.158 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,158 seastar - Reactor backend: linux-aio 2023-10-31T23:29:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,173 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T23:29:04.174 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,173 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T23:29:04.175 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,173 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T23:29:04.175 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,174 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T23:29:04.176 INFO:teuthology.orchestra.run.smithi050.stderr:==79899==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T23:29:04.182 INFO:teuthology.orchestra.run.smithi050.stderr:==79899==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcd192e000; bottom 0x7f0feabeb000; size: 0x00ece6d43000 (1017484947456) 2023-10-31T23:29:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:False positive error reports may follow 2023-10-31T23:29:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T23:29:04.208 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x22aa7d15 0x22aa9006 0x22a84779 0x22a84e30 0x22a9952b 0x22a9adb3 0x22ae4cc4 0x22aae595 0x22ab1bbe 0x22ac067f 0x22ae0321 0x22ae0ba9 0x22a93ca9 0x22aa32c9 0x22aa49c5 0x22aa539e 0x22aa6489 0x22bbd8d8 0x22c5122d 0x229b6079 0x22a8ffff 0x22a92428 0x22a7eb74 0x22a7e1cc 0x22920b9e 0x247b7df3 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1ec5c758 0x1ec668c5 0x1eae33c9 0x3feaf 0x3ff5f 0x1e59ba04 2023-10-31T23:29:04.209 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:04.718 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:04.718 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2023-10-31T23:29:04.718 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2023-10-31T23:29:04.718 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2023-10-31T23:29:04.719 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:04.719 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T23:29:04.771 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,771 seastar - Reactor backend: linux-aio 2023-10-31T23:29:04.789 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,789 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T23:29:04.790 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,789 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T23:29:04.790 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,789 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T23:29:04.797 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:04,797 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T23:29:04.799 INFO:teuthology.orchestra.run.smithi050.stderr:==79897==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T23:29:04.805 INFO:teuthology.orchestra.run.smithi050.stderr:==79897==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcd192e000; bottom 0x7f0fe89d4000; size: 0x00ece8f5a000 (1017520693248) 2023-10-31T23:29:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:False positive error reports may follow 2023-10-31T23:29:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T23:29:04.831 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x228b1e48 0x228b6ef6 0x247b8898 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1eae4c7f 0x3feaf 0x3ff5f 0x1e59ba04 2023-10-31T23:29:04.832 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:04.865 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x9ff52 0x58849 0x2bd7ca2c 0x2bddce8a 0x273d25f0 0x25e822cc 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-10-31T23:29:04.865 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9a3004d0 0xffffffff9a30dd2b 0xffffffff9a313884 0xffffffff9a104e95 0xffffffff9a10604b 0xffffffff9a10664d 0xffffffff9ac5e18c 0xffffffff9ae000aa 2023-10-31T23:29:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 90 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2604eb84 0x260527fc 0x26055843 0x25e854b7 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-10-31T23:29:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9a14ffa3 0xffffffff9a14ffea 0xffffffff9ac5e18c 0xffffffff9ae000aa 2023-10-31T23:29:08.847 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:08.847 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2023-10-31T23:29:08.847 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2023-10-31T23:29:08.848 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2023-10-31T23:29:08.848 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:08.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:08.848 INFO:teuthology.orchestra.run.smithi050.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a 2023-10-31T23:29:08.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-31T23:29:08.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:29:09.471 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T23:29:09.472 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:09,472 seastar - Reactor backend: linux-aio 2023-10-31T23:29:09.493 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:09,492 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T23:29:09.493 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:09,492 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T23:29:09.493 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:09,492 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T23:29:09.494 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:09,492 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T23:29:09.495 INFO:teuthology.orchestra.run.smithi050.stderr:==80081==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T23:29:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:==80081==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffddbc97000; bottom 0x7f6c17c5c000; size: 0x0091c403b000 (626058833920) 2023-10-31T23:29:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:False positive error reports may follow 2023-10-31T23:29:09.502 INFO:teuthology.orchestra.run.smithi050.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T23:29:09.527 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x2bd96 0x2c061 0x2c0f0 0x2c3b8 0x2c5a7 0x2857c 0xb6323 0x22891ac7 0x22890118 0x22b8a272 0x22b8cbce 0x22bbd001 0x22c5122d 0x229b6079 0x22a8ffff 0x22a92428 0x22a7eb74 0x22a7e1cc 0x22920b9e 0x247b7df3 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1ec5c758 0x1ec668c5 0x1eae33c9 0x3feaf 0x3ff5f 0x1e59ba04 2023-10-31T23:29:09.528 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:10.035 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:10.035 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2023-10-31T23:29:10.035 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2023-10-31T23:29:10.035 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2023-10-31T23:29:10.036 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:10.036 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:10.091 INFO:teuthology.orchestra.run.smithi050.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T23:29:10.092 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:10,091 seastar - Reactor backend: linux-aio 2023-10-31T23:29:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:10,109 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T23:29:10.109 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:10,109 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T23:29:10.110 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:10,109 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T23:29:10.117 INFO:teuthology.orchestra.run.smithi050.stderr:INFO 2023-10-31 23:29:10,117 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T23:29:10.119 INFO:teuthology.orchestra.run.smithi050.stderr:==80079==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T23:29:10.125 INFO:teuthology.orchestra.run.smithi050.stderr:==80079==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffddbc97000; bottom 0x7f6c15a45000; size: 0x0091c6252000 (626094579712) 2023-10-31T23:29:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:False positive error reports may follow 2023-10-31T23:29:10.126 INFO:teuthology.orchestra.run.smithi050.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T23:29:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x39787 0x94456 0x94d46 0x227ba669 0x228b71f1 0x247b8898 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1eae4c7f 0x3feaf 0x3ff5f 0x1e59ba04 2023-10-31T23:29:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:10.184 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x273489fe 0x2734a307 0x273ac9f8 0x272607f0 0x272dd1f7 0x25e81878 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-10-31T23:29:10.185 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:10.243 INFO:teuthology.orchestra.run.smithi050.stderr:Reactor stalled for 90 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2604eb84 0x260527fc 0x26055843 0x25e854b7 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-10-31T23:29:10.243 INFO:teuthology.orchestra.run.smithi050.stderr:kernel callstack: 2023-10-31T23:29:14.178 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:14.178 INFO:teuthology.orchestra.run.smithi050.stderr:Suppressions used: 2023-10-31T23:29:14.178 INFO:teuthology.orchestra.run.smithi050.stderr: count bytes template 2023-10-31T23:29:14.179 INFO:teuthology.orchestra.run.smithi050.stderr: 1 8 MallocExtension::Register 2023-10-31T23:29:14.179 INFO:teuthology.orchestra.run.smithi050.stderr:----------------------------------------------------- 2023-10-31T23:29:14.179 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:14.179 INFO:teuthology.orchestra.run.smithi050.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 470ce34b-f6e9-4b2a-8c22-e701db0cdb0a 2023-10-31T23:29:14.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-31T23:29:14.245 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-31T23:29:14.246 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:14.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-31T23:29:14.316 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:14.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-31T23:29:14.387 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:14.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-31T23:29:14.457 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:14.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-31T23:29:14.527 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:14.528 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-31T23:29:14.582 INFO:tasks.ceph:Adding keys to all mons... 2023-10-31T23:29:14.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-31T23:29:14.650 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2023-10-31T23:29:14.651 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQC5jUFlNM0tChAA3SY/9pZIBQvE8rvddKE5+A== 2023-10-31T23:29:14.651 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2023-10-31T23:29:14.651 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQC7jUFluNO/HxAA7ixFkw7eVNqUGdsbImVUcw== 2023-10-31T23:29:14.651 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2023-10-31T23:29:14.652 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDAjUFlNw/vMhAAyySunl5x/lpnJw7e2wvzgA== 2023-10-31T23:29:14.652 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2023-10-31T23:29:14.652 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDGjUFl+S5pChAAHNNf/9lTE0KNxJ1LtVxzcw== 2023-10-31T23:29:14.652 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2023-10-31T23:29:14.652 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQC5jUFlM3J4DRAAB3XK//3qIhIf3fj7R/lfEg== 2023-10-31T23:29:14.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-31T23:29:14.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T23:29:14.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T23:29:14.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T23:29:14.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-31T23:29:15.004 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-31T23:29:15.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-31T23:29:15.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-31T23:29:15.143 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:15.142+0000 7f8bfaf6c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.144 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:15.142+0000 7f8bfaf6c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-31T23:29:15.190 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-31T23:29:15.245 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-31T23:29:15.245 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-31T23:29:15.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-31T23:29:15.287 INFO:tasks.ceph.mon.a:Started 2023-10-31T23:29:15.288 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-31T23:29:15.288 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-31T23:29:15.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-31T23:29:15.290 INFO:tasks.ceph.mgr.x:Started 2023-10-31T23:29:15.291 INFO:tasks.ceph:Setting crush tunables to default 2023-10-31T23:29:15.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2023-10-31T23:29:15.374 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:15.372+0000 7f0c78017f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-31T23:29:15.373+0000 7fb79b82c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-31T23:29:15.373+0000 7fb79b82c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.375 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-31T23:29:15.373+0000 7fb79b82c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.404 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:15.398+0000 7f712899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.425 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:15.423+0000 7f712899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.496 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2023-10-31T23:29:15.507 INFO:tasks.ceph:check_enable_crimson: True 2023-10-31T23:29:15.508 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-31T23:29:15.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-31T23:29:15.577 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:15.575+0000 7f0c78017f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.577 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:15.576+0000 7f0c78017f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.617 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:15.612+0000 7f581d91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:15.618+0000 7f581d91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:15.699 INFO:tasks.ceph.mon.a.smithi050.stderr:2023-10-31T23:29:15.698+0000 7fb792bf1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-31T23:29:15.700 INFO:tasks.ceph.mon.a.smithi050.stderr:Please be aware that this feature is experimental, untested, 2023-10-31T23:29:15.700 INFO:tasks.ceph.mon.a.smithi050.stderr:unsupported, and may result in data corruption, data loss, 2023-10-31T23:29:15.700 INFO:tasks.ceph.mon.a.smithi050.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-31T23:29:15.700 INFO:tasks.ceph.mon.a.smithi050.stderr:feature with important data. 2023-10-31T23:29:15.700 INFO:tasks.ceph.mon.a.smithi050.stderr: 2023-10-31T23:29:16.108 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.106+0000 7f0c78017f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-31T23:29:16.162 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.160+0000 7f0c78017f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-31T23:29:16.508 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-31T23:29:16.509 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:16.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-31T23:29:16.539 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:16.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-31T23:29:16.610 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:16.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-31T23:29:16.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new a57e578c-836f-4c29-bda4-527b2d14887b 0 2023-10-31T23:29:16.718 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.716+0000 7f0c78017f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-31T23:29:16.774 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.773+0000 7f0c78017f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-31T23:29:16.824 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.822+0000 7f0c78017f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-31T23:29:16.828 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:16.822+0000 7fba1bb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:16.830 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:16.829+0000 7fba1bb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:16.881 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.879+0000 7f0c78017f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-31T23:29:16.908 INFO:teuthology.orchestra.run.smithi050.stdout:0 2023-10-31T23:29:16.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 723a3029-4202-42ec-9be3-38a318b6b915 1 2023-10-31T23:29:16.927 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:16.926+0000 7f0c78017f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-31T23:29:17.030 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:17.023+0000 7ff2f8c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:17.031 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:17.030+0000 7ff2f8c13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:17.036 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:17.035+0000 7f0c78017f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-31T23:29:17.109 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-10-31T23:29:17.112 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:17.110+0000 7f0c78017f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-31T23:29:17.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 37be6e9b-e043-4df8-8264-def2b3f78610 2 2023-10-31T23:29:17.160 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:17.159+0000 7f0c78017f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-31T23:29:17.216 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:17.215+0000 7f0c78017f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-31T23:29:17.268 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:17.266+0000 7f0c78017f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-31T23:29:17.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:17.264+0000 7efc868ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:17.273 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:17.271+0000 7efc868ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:17.370 INFO:teuthology.orchestra.run.smithi050.stdout:2 2023-10-31T23:29:17.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-31T23:29:17.416 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-31T23:29:17.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-31T23:29:17.458 INFO:tasks.ceph.osd.0:Started 2023-10-31T23:29:17.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-31T23:29:17.494 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-31T23:29:17.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-31T23:29:17.536 INFO:tasks.ceph.osd.1:Started 2023-10-31T23:29:17.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-31T23:29:17.575 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-31T23:29:17.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-31T23:29:17.617 INFO:tasks.ceph.osd.2:Started 2023-10-31T23:29:17.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T23:29:17.760 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:17.759+0000 7f0c78017f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-31T23:29:17.778 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:17.772+0000 7fd596ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:17.781 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:17.779+0000 7fd596ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:17.931 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:17.932 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":6,"fsid":"470ce34b-f6e9-4b2a-8c22-e701db0cdb0a","created":"2023-10-31T23:29:15.416452+0000","modified":"2023-10-31T23:29:17.369068+0000","last_up_change":"0.000000","last_in_change":"2023-10-31T23:29:17.369068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a57e578c-836f-4c29-bda4-527b2d14887b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"723a3029-4202-42ec-9be3-38a318b6b915","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"37be6e9b-e043-4df8-8264-def2b3f78610","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T23:29:17.957 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-31T23:29:17.958 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-31T23:29:18.002 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.000+0000 7f0c78017f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-31T23:29:18.129 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.127+0000 7f0c78017f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-31T23:29:18.181 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.180+0000 7f0c78017f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-31T23:29:18.261 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-31T23:29:18.289 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.288+0000 7f0c78017f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-31T23:29:18.362 INFO:teuthology.misc.health.smithi050.stderr:2023-10-31T23:29:18.357+0000 7f22ee11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:18.363 INFO:teuthology.misc.health.smithi050.stderr:2023-10-31T23:29:18.362+0000 7f22ee11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:18.398 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.396+0000 7f0c78017f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-31T23:29:18.431 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-31T23:29:18.431 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":6,"fsid":"470ce34b-f6e9-4b2a-8c22-e701db0cdb0a","created":"2023-10-31T23:29:15.416452+0000","modified":"2023-10-31T23:29:17.369068+0000","last_up_change":"0.000000","last_in_change":"2023-10-31T23:29:17.369068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a57e578c-836f-4c29-bda4-527b2d14887b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"723a3029-4202-42ec-9be3-38a318b6b915","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"37be6e9b-e043-4df8-8264-def2b3f78610","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T23:29:18.442 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-31T23:29:18.555 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.554+0000 7f0c78017f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-31T23:29:18.603 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.602+0000 7f0c78017f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-31T23:29:18.759 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:18.757+0000 7f0c78017f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-31T23:29:19.632 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:19.630+0000 7f0c78017f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-31T23:29:19.923 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:19.922+0000 7f0c78017f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-31T23:29:19.978 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:19.977+0000 7f0c78017f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-31T23:29:20.086 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:20.085+0000 7f0c78017f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-31T23:29:21.107 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-10-31T23:29:21.105+0000 7f0c182ab640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-31T23:29:24.744 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-31T23:29:24.855 INFO:teuthology.misc.health.smithi050.stderr:2023-10-31T23:29:24.851+0000 7fd48b55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:24.857 INFO:teuthology.misc.health.smithi050.stderr:2023-10-31T23:29:24.856+0000 7fd48b55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:25.043 INFO:teuthology.misc.health.smithi050.stdout: 2023-10-31T23:29:25.043 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":8,"fsid":"470ce34b-f6e9-4b2a-8c22-e701db0cdb0a","created":"2023-10-31T23:29:15.416452+0000","modified":"2023-10-31T23:29:23.106669+0000","last_up_change":"2023-10-31T23:29:23.106669+0000","last_in_change":"2023-10-31T23:29:17.369068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a57e578c-836f-4c29-bda4-527b2d14887b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3363552748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3363552748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":3363552748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3363552748}]},"public_addr":"172.21.15.50:6805/3363552748","cluster_addr":"172.21.15.50:6806/3363552748","heartbeat_back_addr":"172.21.15.50:6813/3363552748","heartbeat_front_addr":"172.21.15.50:6812/3363552748","state":["exists","up"]},{"osd":1,"uuid":"723a3029-4202-42ec-9be3-38a318b6b915","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1940359164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1940359164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":1940359164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1940359164}]},"public_addr":"172.21.15.50:6803/1940359164","cluster_addr":"172.21.15.50:6802/1940359164","heartbeat_back_addr":"172.21.15.50:6811/1940359164","heartbeat_front_addr":"172.21.15.50:6810/1940359164","state":["exists","up"]},{"osd":2,"uuid":"37be6e9b-e043-4df8-8264-def2b3f78610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2781001542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2781001542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2781001542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2781001542}]},"public_addr":"172.21.15.50:6804/2781001542","cluster_addr":"172.21.15.50:6807/2781001542","heartbeat_back_addr":"172.21.15.50:6808/2781001542","heartbeat_front_addr":"172.21.15.50:6809/2781001542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T23:29:25.053 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-31T23:29:25.054 INFO:tasks.ceph:Creating RBD pool 2023-10-31T23:29:25.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-31T23:29:25.167 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:25.160+0000 7f2e72708640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:25.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:25.168+0000 7f2e72708640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:26.116 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2023-10-31T23:29:26.134 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-31T23:29:26.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:26.258+0000 7f0b3b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:26.269+0000 7f0b3b42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-31T23:29:27.134 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-31T23:29:27.134 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-31T23:29:27.135 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-31T23:29:27.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-31T23:29:27.136 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-31T23:29:27.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:27.264+0000 7f059bd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:27.270 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:27.269+0000 7f059bd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:27.505 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:27.516 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":937850819},{"type":"v1","addr":"172.21.15.50:6801","nonce":937850819}]},"active_addr":"172.21.15.50:6801/937850819","active_change":"2023-10-31T23:29:20.098998+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1691046234}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":222466461}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2291130252}]}]} 2023-10-31T23:29:27.519 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-31T23:29:27.519 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-31T23:29:27.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T23:29:27.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:27.626+0000 7f26ce95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:27.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:27.631+0000 7f26ce95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:27.819 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":11,"fsid":"470ce34b-f6e9-4b2a-8c22-e701db0cdb0a","created":"2023-10-31T23:29:15.416452+0000","modified":"2023-10-31T23:29:27.119166+0000","last_up_change":"2023-10-31T23:29:23.106669+0000","last_in_change":"2023-10-31T23:29:17.369068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-31T23:29:24.157460+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-31T23:29:25.384003+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a57e578c-836f-4c29-bda4-527b2d14887b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3363552748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3363552748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":3363552748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3363552748}]},"public_addr":"172.21.15.50:6805/3363552748","cluster_addr":"172.21.15.50:6806/3363552748","heartbeat_back_addr":"172.21.15.50:6813/3363552748","heartbeat_front_addr":"172.21.15.50:6812/3363552748","state":["exists","up"]},{"osd":1,"uuid":"723a3029-4202-42ec-9be3-38a318b6b915","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1940359164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1940359164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":1940359164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1940359164}]},"public_addr":"172.21.15.50:6803/1940359164","cluster_addr":"172.21.15.50:6802/1940359164","heartbeat_back_addr":"172.21.15.50:6811/1940359164","heartbeat_front_addr":"172.21.15.50:6810/1940359164","state":["exists","up"]},{"osd":2,"uuid":"37be6e9b-e043-4df8-8264-def2b3f78610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2781001542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2781001542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2781001542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2781001542}]},"public_addr":"172.21.15.50:6804/2781001542","cluster_addr":"172.21.15.50:6807/2781001542","heartbeat_back_addr":"172.21.15.50:6808/2781001542","heartbeat_front_addr":"172.21.15.50:6809/2781001542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T23:29:27.831 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-31T23:29:27.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T23:29:27.943 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:27.939+0000 7fee3ca64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:27.993 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:27.944+0000 7fee3ca64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.129 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:28.129 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":12,"fsid":"470ce34b-f6e9-4b2a-8c22-e701db0cdb0a","created":"2023-10-31T23:29:15.416452+0000","modified":"2023-10-31T23:29:28.123082+0000","last_up_change":"2023-10-31T23:29:23.106669+0000","last_in_change":"2023-10-31T23:29:17.369068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-31T23:29:24.157460+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-31T23:29:25.384003+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a57e578c-836f-4c29-bda4-527b2d14887b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3363552748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3363552748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":3363552748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3363552748}]},"public_addr":"172.21.15.50:6805/3363552748","cluster_addr":"172.21.15.50:6806/3363552748","heartbeat_back_addr":"172.21.15.50:6813/3363552748","heartbeat_front_addr":"172.21.15.50:6812/3363552748","state":["exists","up"]},{"osd":1,"uuid":"723a3029-4202-42ec-9be3-38a318b6b915","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1940359164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1940359164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":1940359164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":1940359164}]},"public_addr":"172.21.15.50:6803/1940359164","cluster_addr":"172.21.15.50:6802/1940359164","heartbeat_back_addr":"172.21.15.50:6811/1940359164","heartbeat_front_addr":"172.21.15.50:6810/1940359164","state":["exists","up"]},{"osd":2,"uuid":"37be6e9b-e043-4df8-8264-def2b3f78610","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2781001542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2781001542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2781001542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2781001542}]},"public_addr":"172.21.15.50:6804/2781001542","cluster_addr":"172.21.15.50:6807/2781001542","heartbeat_back_addr":"172.21.15.50:6808/2781001542","heartbeat_front_addr":"172.21.15.50:6809/2781001542","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T23:29:28.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-31T23:29:28.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-31T23:29:28.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-31T23:29:28.265 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.260+0000 7f4728a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.260+0000 7ff39d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.266 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.261+0000 7efd8ee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.266+0000 7ff39d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.268 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.266+0000 7f4728a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.269 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.267+0000 7efd8ee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.293 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-31T23:29:28.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-31T23:29:28.296 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-31T23:29:28.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-31T23:29:28.297 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-31T23:29:28.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-31T23:29:28.406 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.401+0000 7f214bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.408 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.407+0000 7f214bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.452 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.447+0000 7f0040d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.453 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.447+0000 7f9305a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.454 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.453+0000 7f9305a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.455 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.453+0000 7f0040d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.595 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-31T23:29:28.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-31T23:29:28.608 DEBUG:teuthology.parallel:result is None 2023-10-31T23:29:28.647 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-31T23:29:28.649 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2023-10-31T23:29:28.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-31T23:29:28.660 DEBUG:teuthology.parallel:result is None 2023-10-31T23:29:28.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-31T23:29:28.662 DEBUG:teuthology.parallel:result is None 2023-10-31T23:29:28.662 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-31T23:29:28.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-31T23:29:28.813 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.808+0000 7fbe3d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:28.813+0000 7fbe3d46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:28.999 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:28.999 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-31T23:29:29.010 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-31T23:29:28.281796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59816,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.225794+0000","last_change":"2023-10-31T23:29:27.225878+0000","last_active":"2023-10-31T23:29:27.225794+0000","last_peered":"2023-10-31T23:29:27.225794+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.223887+0000","last_became_peered":"2023-10-31T23:29:27.223887+0000","last_unstale":"2023-10-31T23:29:27.225794+0000","last_undegraded":"2023-10-31T23:29:27.225794+0000","last_fullsized":"2023-10-31T23:29:27.225794+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.262875+0000","last_change":"2023-10-31T23:29:27.262929+0000","last_active":"2023-10-31T23:29:27.262875+0000","last_peered":"2023-10-31T23:29:27.262875+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.261124+0000","last_became_peered":"2023-10-31T23:29:27.261124+0000","last_unstale":"2023-10-31T23:29:27.262875+0000","last_undegraded":"2023-10-31T23:29:27.262875+0000","last_fullsized":"2023-10-31T23:29:27.262875+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.244753+0000","last_change":"2023-10-31T23:29:27.244827+0000","last_active":"2023-10-31T23:29:27.244753+0000","last_peered":"2023-10-31T23:29:27.244753+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.242565+0000","last_became_peered":"2023-10-31T23:29:27.242565+0000","last_unstale":"2023-10-31T23:29:27.244753+0000","last_undegraded":"2023-10-31T23:29:27.244753+0000","last_fullsized":"2023-10-31T23:29:27.244753+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.237051+0000","last_change":"2023-10-31T23:29:27.237195+0000","last_active":"2023-10-31T23:29:27.237051+0000","last_peered":"2023-10-31T23:29:27.237051+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.227900+0000","last_became_peered":"2023-10-31T23:29:27.227900+0000","last_unstale":"2023-10-31T23:29:27.237051+0000","last_undegraded":"2023-10-31T23:29:27.237051+0000","last_fullsized":"2023-10-31T23:29:27.237051+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T23:29:28.167029+0000","last_change":"2023-10-31T23:29:27.196822+0000","last_active":"2023-10-31T23:29:28.167029+0000","last_peered":"2023-10-31T23:29:28.167029+0000","last_clean":"2023-10-31T23:29:28.167029+0000","last_became_active":"2023-10-31T23:29:27.194335+0000","last_became_peered":"2023-10-31T23:29:27.194335+0000","last_unstale":"2023-10-31T23:29:28.167029+0000","last_undegraded":"2023-10-31T23:29:28.167029+0000","last_fullsized":"2023-10-31T23:29:28.167029+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T23:29:28.131404+0000","last_change":"2023-10-31T23:29:27.217919+0000","last_active":"2023-10-31T23:29:28.131404+0000","last_peered":"2023-10-31T23:29:28.131404+0000","last_clean":"2023-10-31T23:29:28.131404+0000","last_became_active":"2023-10-31T23:29:27.215641+0000","last_became_peered":"2023-10-31T23:29:27.215641+0000","last_unstale":"2023-10-31T23:29:28.131404+0000","last_undegraded":"2023-10-31T23:29:28.131404+0000","last_fullsized":"2023-10-31T23:29:28.131404+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T23:29:28.130848+0000","last_change":"2023-10-31T23:29:27.201563+0000","last_active":"2023-10-31T23:29:28.130848+0000","last_peered":"2023-10-31T23:29:28.130848+0000","last_clean":"2023-10-31T23:29:28.130848+0000","last_became_active":"2023-10-31T23:29:27.199548+0000","last_became_peered":"2023-10-31T23:29:27.199548+0000","last_unstale":"2023-10-31T23:29:28.130848+0000","last_undegraded":"2023-10-31T23:29:28.130848+0000","last_fullsized":"2023-10-31T23:29:28.130848+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.202714+0000","last_change":"2023-10-31T23:29:27.202778+0000","last_active":"2023-10-31T23:29:27.202714+0000","last_peered":"2023-10-31T23:29:27.202714+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.199583+0000","last_became_peered":"2023-10-31T23:29:27.199583+0000","last_unstale":"2023-10-31T23:29:27.202714+0000","last_undegraded":"2023-10-31T23:29:27.202714+0000","last_fullsized":"2023-10-31T23:29:27.202714+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.277172+0000","last_change":"2023-10-31T23:29:26.190682+0000","last_active":"2023-10-31T23:29:27.277172+0000","last_peered":"2023-10-31T23:29:27.277172+0000","last_clean":"2023-10-31T23:29:27.277172+0000","last_became_active":"2023-10-31T23:29:26.182517+0000","last_became_peered":"2023-10-31T23:29:26.182517+0000","last_unstale":"2023-10-31T23:29:27.277172+0000","last_undegraded":"2023-10-31T23:29:27.277172+0000","last_fullsized":"2023-10-31T23:29:27.277172+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:25.111725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:25.111725+0000","last_clean_scrub_stamp":"2023-10-31T23:29:25.111725+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29908,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29908,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-31T23:29:29.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-31T23:29:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:29.120+0000 7f7e33304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:29.127 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:29.126+0000 7f7e33304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:29.310 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:29.310 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2023-10-31T23:29:29.321 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-31T23:29:28.281796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59816,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.225794+0000","last_change":"2023-10-31T23:29:27.225878+0000","last_active":"2023-10-31T23:29:27.225794+0000","last_peered":"2023-10-31T23:29:27.225794+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.223887+0000","last_became_peered":"2023-10-31T23:29:27.223887+0000","last_unstale":"2023-10-31T23:29:27.225794+0000","last_undegraded":"2023-10-31T23:29:27.225794+0000","last_fullsized":"2023-10-31T23:29:27.225794+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.262875+0000","last_change":"2023-10-31T23:29:27.262929+0000","last_active":"2023-10-31T23:29:27.262875+0000","last_peered":"2023-10-31T23:29:27.262875+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.261124+0000","last_became_peered":"2023-10-31T23:29:27.261124+0000","last_unstale":"2023-10-31T23:29:27.262875+0000","last_undegraded":"2023-10-31T23:29:27.262875+0000","last_fullsized":"2023-10-31T23:29:27.262875+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.244753+0000","last_change":"2023-10-31T23:29:27.244827+0000","last_active":"2023-10-31T23:29:27.244753+0000","last_peered":"2023-10-31T23:29:27.244753+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.242565+0000","last_became_peered":"2023-10-31T23:29:27.242565+0000","last_unstale":"2023-10-31T23:29:27.244753+0000","last_undegraded":"2023-10-31T23:29:27.244753+0000","last_fullsized":"2023-10-31T23:29:27.244753+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.237051+0000","last_change":"2023-10-31T23:29:27.237195+0000","last_active":"2023-10-31T23:29:27.237051+0000","last_peered":"2023-10-31T23:29:27.237051+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.227900+0000","last_became_peered":"2023-10-31T23:29:27.227900+0000","last_unstale":"2023-10-31T23:29:27.237051+0000","last_undegraded":"2023-10-31T23:29:27.237051+0000","last_fullsized":"2023-10-31T23:29:27.237051+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T23:29:28.167029+0000","last_change":"2023-10-31T23:29:27.196822+0000","last_active":"2023-10-31T23:29:28.167029+0000","last_peered":"2023-10-31T23:29:28.167029+0000","last_clean":"2023-10-31T23:29:28.167029+0000","last_became_active":"2023-10-31T23:29:27.194335+0000","last_became_peered":"2023-10-31T23:29:27.194335+0000","last_unstale":"2023-10-31T23:29:28.167029+0000","last_undegraded":"2023-10-31T23:29:28.167029+0000","last_fullsized":"2023-10-31T23:29:28.167029+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T23:29:28.131404+0000","last_change":"2023-10-31T23:29:27.217919+0000","last_active":"2023-10-31T23:29:28.131404+0000","last_peered":"2023-10-31T23:29:28.131404+0000","last_clean":"2023-10-31T23:29:28.131404+0000","last_became_active":"2023-10-31T23:29:27.215641+0000","last_became_peered":"2023-10-31T23:29:27.215641+0000","last_unstale":"2023-10-31T23:29:28.131404+0000","last_undegraded":"2023-10-31T23:29:28.131404+0000","last_fullsized":"2023-10-31T23:29:28.131404+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T23:29:28.130848+0000","last_change":"2023-10-31T23:29:27.201563+0000","last_active":"2023-10-31T23:29:28.130848+0000","last_peered":"2023-10-31T23:29:28.130848+0000","last_clean":"2023-10-31T23:29:28.130848+0000","last_became_active":"2023-10-31T23:29:27.199548+0000","last_became_peered":"2023-10-31T23:29:27.199548+0000","last_unstale":"2023-10-31T23:29:28.130848+0000","last_undegraded":"2023-10-31T23:29:28.130848+0000","last_fullsized":"2023-10-31T23:29:28.130848+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.202714+0000","last_change":"2023-10-31T23:29:27.202778+0000","last_active":"2023-10-31T23:29:27.202714+0000","last_peered":"2023-10-31T23:29:27.202714+0000","last_clean":"2023-10-31T23:29:26.114636+0000","last_became_active":"2023-10-31T23:29:27.199583+0000","last_became_peered":"2023-10-31T23:29:27.199583+0000","last_unstale":"2023-10-31T23:29:27.202714+0000","last_undegraded":"2023-10-31T23:29:27.202714+0000","last_fullsized":"2023-10-31T23:29:27.202714+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:26.114636+0000","last_clean_scrub_stamp":"2023-10-31T23:29:26.114636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-31T23:29:27.277172+0000","last_change":"2023-10-31T23:29:26.190682+0000","last_active":"2023-10-31T23:29:27.277172+0000","last_peered":"2023-10-31T23:29:27.277172+0000","last_clean":"2023-10-31T23:29:27.277172+0000","last_became_active":"2023-10-31T23:29:26.182517+0000","last_became_peered":"2023-10-31T23:29:26.182517+0000","last_unstale":"2023-10-31T23:29:27.277172+0000","last_undegraded":"2023-10-31T23:29:27.277172+0000","last_fullsized":"2023-10-31T23:29:27.277172+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T23:29:25.111725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T23:29:25.111725+0000","last_clean_scrub_stamp":"2023-10-31T23:29:25.111725+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29908,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29908,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-31T23:29:29.322 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-31T23:29:29.323 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-31T23:29:29.323 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-31T23:29:29.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-31T23:29:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:29.431+0000 7fe879b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:29:29.434+0000 7fe879b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:29:29.655 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:29.655 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-31T23:29:29.667 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-31T23:29:29.667 INFO:teuthology.run_tasks:Running task cbt... 2023-10-31T23:29:29.682 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi050.front.sepia.ceph.com', 'osds': ['ubuntu@smithi050.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi050.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi050.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi050.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-10-31T23:29:29.683 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-10-31T23:29:29.700 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:29:29.701 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-31T23:29:29.754 DEBUG:teuthology.orchestra.run.smithi050:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-10-31T23:29:29.809 INFO:teuthology.orchestra.run.smithi050.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-10-31T23:29:30.311 INFO:tasks.cbt:Installing collectl 2023-10-31T23:29:30.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-10-31T23:29:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:--2023-10-31 23:29:30-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-10-31T23:29:30.561 INFO:teuthology.orchestra.run.smithi050.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2023-10-31T23:29:30.571 INFO:teuthology.orchestra.run.smithi050.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-10-31T23:29:30.802 INFO:teuthology.orchestra.run.smithi050.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-31T23:29:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQY3aFq3wpZvQt8Rt_LFvUirwaV0KkyjdwLFdWBqJXSq12brazc3Qt8Og58flsLR7eNSfqr7sNJ_tsoGc_5CUkGQp6w%3D%3D&use_mirror=versaweb&r= [following] 2023-10-31T23:29:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:--2023-10-31 23:29:30-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQY3aFq3wpZvQt8Rt_LFvUirwaV0KkyjdwLFdWBqJXSq12brazc3Qt8Og58flsLR7eNSfqr7sNJ_tsoGc_5CUkGQp6w%3D%3D&use_mirror=versaweb&r= 2023-10-31T23:29:30.803 INFO:teuthology.orchestra.run.smithi050.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-10-31T23:29:30.874 INFO:teuthology.orchestra.run.smithi050.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-10-31T23:29:31.113 INFO:teuthology.orchestra.run.smithi050.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-31T23:29:31.114 INFO:teuthology.orchestra.run.smithi050.stderr:Location: https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-10-31T23:29:31.114 INFO:teuthology.orchestra.run.smithi050.stderr:--2023-10-31 23:29:31-- https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-10-31T23:29:31.123 INFO:teuthology.orchestra.run.smithi050.stderr:Resolving versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)... 162.251.232.173 2023-10-31T23:29:31.194 INFO:teuthology.orchestra.run.smithi050.stderr:Connecting to versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)|162.251.232.173|:443... connected. 2023-10-31T23:29:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-31T23:29:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:Length: 620693 (606K) [application/x-gzip] 2023-10-31T23:29:31.590 INFO:teuthology.orchestra.run.smithi050.stderr:Saving to: ‘collectl.tar.gz’ 2023-10-31T23:29:31.591 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:31.660 INFO:teuthology.orchestra.run.smithi050.stderr: 0K .......... .......... .......... .......... .......... 8% 701K 1s 2023-10-31T23:29:31.729 INFO:teuthology.orchestra.run.smithi050.stderr: 50K .......... .......... .......... .......... .......... 16% 734K 1s 2023-10-31T23:29:31.732 INFO:teuthology.orchestra.run.smithi050.stderr: 100K .......... .......... .......... .......... .......... 24% 13.3M 0s 2023-10-31T23:29:31.799 INFO:teuthology.orchestra.run.smithi050.stderr: 150K .......... .......... .......... .......... .......... 32% 759K 0s 2023-10-31T23:29:31.800 INFO:teuthology.orchestra.run.smithi050.stderr: 200K .......... .......... .......... .......... .......... 41% 59.4M 0s 2023-10-31T23:29:31.800 INFO:teuthology.orchestra.run.smithi050.stderr: 250K .......... .......... .......... .......... .......... 49% 33.7M 0s 2023-10-31T23:29:31.804 INFO:teuthology.orchestra.run.smithi050.stderr: 300K .......... .......... .......... .......... .......... 57% 14.1M 0s 2023-10-31T23:29:31.811 INFO:teuthology.orchestra.run.smithi050.stderr: 350K .......... .......... .......... .......... .......... 65% 6.96M 0s 2023-10-31T23:29:31.873 INFO:teuthology.orchestra.run.smithi050.stderr: 400K .......... .......... .......... .......... .......... 74% 804K 0s 2023-10-31T23:29:31.876 INFO:teuthology.orchestra.run.smithi050.stderr: 450K .......... .......... .......... .......... .......... 82% 34.3M 0s 2023-10-31T23:29:31.876 INFO:teuthology.orchestra.run.smithi050.stderr: 500K .......... .......... .......... .......... .......... 90% 92.6M 0s 2023-10-31T23:29:31.876 INFO:teuthology.orchestra.run.smithi050.stderr: 550K .......... .......... .......... .......... .......... 98% 86.2M 0s 2023-10-31T23:29:31.877 INFO:teuthology.orchestra.run.smithi050.stderr: 600K ...... 100% 11.4T=0.3s 2023-10-31T23:29:31.877 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:31.877 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31 23:29:31 (2.07 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-10-31T23:29:31.878 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:29:31.927 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/ 2023-10-31T23:29:31.927 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/collectl.conf 2023-10-31T23:29:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/ 2023-10-31T23:29:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/SlowProc.html 2023-10-31T23:29:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-10-31T23:29:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/WhySummary.html 2023-10-31T23:29:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Socket.html 2023-10-31T23:29:31.928 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/colmux.html 2023-10-31T23:29:31.929 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Tutorial.html 2023-10-31T23:29:31.929 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Gexpr.html 2023-10-31T23:29:31.929 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Import.html 2023-10-31T23:29:31.929 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/style.css 2023-10-31T23:29:31.929 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-10-31T23:29:31.930 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-10-31T23:29:31.930 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/FileNaming.html 2023-10-31T23:29:31.930 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/TheMath.html 2023-10-31T23:29:31.930 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Environmental.html 2023-10-31T23:29:31.930 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Process.html 2023-10-31T23:29:31.931 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-10-31T23:29:31.931 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/HiResTime.html 2023-10-31T23:29:31.931 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-10-31T23:29:31.931 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/slub.jpg 2023-10-31T23:29:31.931 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Lustre.html 2023-10-31T23:29:31.932 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Documentation.html 2023-10-31T23:29:31.932 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Inodes.html 2023-10-31T23:29:31.932 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Graphite.html 2023-10-31T23:29:31.932 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/InputFiles.html 2023-10-31T23:29:31.932 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Export.html 2023-10-31T23:29:31.933 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Interrupts.html 2023-10-31T23:29:31.933 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Startup.html 2023-10-31T23:29:31.933 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Performance.html 2023-10-31T23:29:31.933 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Messages.html 2023-10-31T23:29:31.933 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Logging.html 2023-10-31T23:29:31.934 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Data-detail.html 2023-10-31T23:29:31.934 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/CPUs.html 2023-10-31T23:29:31.934 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Matrix.html 2023-10-31T23:29:31.934 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Data-brief.html 2023-10-31T23:29:31.934 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-10-31T23:29:31.935 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Misc.html 2023-10-31T23:29:31.935 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Examples.html 2023-10-31T23:29:31.935 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/OpenStack.html 2023-10-31T23:29:31.935 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-10-31T23:29:31.935 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-10-31T23:29:31.936 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-10-31T23:29:31.936 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-10-31T23:29:31.936 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Data.html 2023-10-31T23:29:31.936 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Exceptions.html 2023-10-31T23:29:31.936 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-10-31T23:29:31.936 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-10-31T23:29:31.937 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-10-31T23:29:31.937 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-10-31T23:29:31.937 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Playback.html 2023-10-31T23:29:31.937 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Hello.html 2023-10-31T23:29:31.937 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Sockets.html 2023-10-31T23:29:31.938 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Network.html 2023-10-31T23:29:31.938 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Infiniband.html 2023-10-31T23:29:31.938 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Memory.html 2023-10-31T23:29:31.938 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-10-31T23:29:31.938 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/Disks.html 2023-10-31T23:29:31.939 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-10-31T23:29:31.939 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/service/ 2023-10-31T23:29:31.939 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/service/collectl.service 2023-10-31T23:29:31.939 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/initd/ 2023-10-31T23:29:31.939 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/initd/collectl-suse 2023-10-31T23:29:31.940 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/initd/collectl-debian 2023-10-31T23:29:31.940 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/initd/collectl-generic 2023-10-31T23:29:31.940 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/initd/collectl 2023-10-31T23:29:31.940 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/misc.ph 2023-10-31T23:29:31.940 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/gexpr.ph 2023-10-31T23:29:31.941 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/statsd.ph 2023-10-31T23:29:31.941 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/envrules.std 2023-10-31T23:29:31.941 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/hello.ph 2023-10-31T23:29:31.941 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/vmsum.ph 2023-10-31T23:29:31.941 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/proctree.ph 2023-10-31T23:29:31.942 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/README-WINDOWS 2023-10-31T23:29:31.942 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/formatit.ph 2023-10-31T23:29:31.943 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/colmux 2023-10-31T23:29:31.944 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/INSTALL 2023-10-31T23:29:31.944 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/graphite.ph 2023-10-31T23:29:31.944 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/RELEASE-collectl 2023-10-31T23:29:31.944 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/vmstat.ph 2023-10-31T23:29:31.944 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/README 2023-10-31T23:29:31.945 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/man1/ 2023-10-31T23:29:31.945 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/man1/collectl.1 2023-10-31T23:29:31.945 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/man1/colmux.1 2023-10-31T23:29:31.945 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/lexpr.ph 2023-10-31T23:29:31.945 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/client.pl 2023-10-31T23:29:31.946 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/collectl 2023-10-31T23:29:31.946 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/COPYING 2023-10-31T23:29:31.946 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/vnet.ph 2023-10-31T23:29:31.946 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/GPL 2023-10-31T23:29:31.946 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/UNINSTALL 2023-10-31T23:29:31.947 INFO:teuthology.orchestra.run.smithi050.stdout:collectl-4.3.1/ARTISTIC 2023-10-31T23:29:31.994 INFO:teuthology.orchestra.run.smithi050.stderr:./INSTALL: line 90: chkconfig: command not found 2023-10-31T23:29:31.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-10-31T23:29:32.571 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:22 ago on Tue 31 Oct 2023 11:26:10 PM UTC. 2023-10-31T23:29:32.667 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-10-31T23:29:32.670 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-10-31T23:29:32.727 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-10-31T23:29:32.728 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:29:32.728 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-10-31T23:29:32.728 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:29:32.728 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-10-31T23:29:32.729 INFO:teuthology.orchestra.run.smithi050.stdout: librbd-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 23 k 2023-10-31T23:29:32.729 INFO:teuthology.orchestra.run.smithi050.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-10-31T23:29:32.729 INFO:teuthology.orchestra.run.smithi050.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-10-31T23:29:32.729 INFO:teuthology.orchestra.run.smithi050.stdout: perf x86_64 5.14.0-378.el9 appstream 7.3 M 2023-10-31T23:29:32.729 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-10-31T23:29:32.730 INFO:teuthology.orchestra.run.smithi050.stdout: libradospp-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 30 k 2023-10-31T23:29:32.730 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:32.730 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-10-31T23:29:32.730 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-10-31T23:29:32.730 INFO:teuthology.orchestra.run.smithi050.stdout:Install 5 Packages 2023-10-31T23:29:32.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:32.731 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.5 M 2023-10-31T23:29:32.732 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 11 M 2023-10-31T23:29:32.732 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-10-31T23:29:33.347 INFO:teuthology.orchestra.run.smithi050.stdout:(1/5): libradospp-devel-18.0.0-6795.gd1dcabba.e 160 kB/s | 30 kB 00:00 2023-10-31T23:29:33.389 INFO:teuthology.orchestra.run.smithi050.stdout:(2/5): librbd-devel-18.0.0-6795.gd1dcabba.el9.x 100 kB/s | 23 kB 00:00 2023-10-31T23:29:33.556 INFO:teuthology.orchestra.run.smithi050.stdout:(3/5): perf-5.14.0-378.el9.x86_64.rpm 18 MB/s | 7.3 MB 00:00 2023-10-31T23:29:33.581 INFO:teuthology.orchestra.run.smithi050.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 68 kB/s | 13 kB 00:00 2023-10-31T23:29:33.623 INFO:teuthology.orchestra.run.smithi050.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 430 kB/s | 118 kB 00:00 2023-10-31T23:29:33.624 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-10-31T23:29:33.624 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.4 MB/s | 7.5 MB 00:00 2023-10-31T23:29:33.704 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-10-31T23:29:33.726 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-10-31T23:29:33.727 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-10-31T23:29:33.941 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-10-31T23:29:33.942 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-10-31T23:29:34.284 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-10-31T23:29:34.368 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-10-31T23:29:34.406 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-10-31T23:29:34.437 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradospp-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/5 2023-10-31T23:29:34.795 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librbd-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/5 2023-10-31T23:29:34.920 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : perf-5.14.0-378.el9.x86_64 5/5 2023-10-31T23:29:35.862 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: perf-5.14.0-378.el9.x86_64 5/5 2023-10-31T23:29:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradospp-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/5 2023-10-31T23:29:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/5 2023-10-31T23:29:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : perf-5.14.0-378.el9.x86_64 3/5 2023-10-31T23:29:35.863 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-10-31T23:29:36.238 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-10-31T23:29:36.238 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:36.238 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-10-31T23:29:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: libradospp-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:29:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: librbd-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-10-31T23:29:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: pdsh-2.34-7.el9.x86_64 2023-10-31T23:29:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-10-31T23:29:36.239 INFO:teuthology.orchestra.run.smithi050.stdout: perf-5.14.0-378.el9.x86_64 2023-10-31T23:29:36.240 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:36.240 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-10-31T23:29:36.625 INFO:tasks.cbt:benchmark: librbdfio 2023-10-31T23:29:36.626 DEBUG:teuthology.orchestra.run.smithi050:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2023-10-31T23:29:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2023-10-31T23:29:38.726 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest/fio && ./configure && make 2023-10-31T23:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout:Operating system Linux 2023-10-31T23:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout:CPU x86_64 2023-10-31T23:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout:Big endian no 2023-10-31T23:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout:Compiler gcc 2023-10-31T23:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout:Cross compile no 2023-10-31T23:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout:Static build no 2023-10-31T23:29:39.432 INFO:teuthology.orchestra.run.smithi050.stdout:Wordsize 64 2023-10-31T23:29:39.439 INFO:teuthology.orchestra.run.smithi050.stdout:zlib no 2023-10-31T23:29:39.478 INFO:teuthology.orchestra.run.smithi050.stdout:fcntl(F_FULLFSYNC) no 2023-10-31T23:29:39.492 INFO:teuthology.orchestra.run.smithi050.stdout:Linux AIO support no 2023-10-31T23:29:39.493 INFO:teuthology.orchestra.run.smithi050.stdout:Linux AIO support rw flags no 2023-10-31T23:29:39.559 INFO:teuthology.orchestra.run.smithi050.stdout:POSIX AIO support yes 2023-10-31T23:29:39.560 INFO:teuthology.orchestra.run.smithi050.stdout:POSIX AIO support needs -lrt no 2023-10-31T23:29:39.591 INFO:teuthology.orchestra.run.smithi050.stdout:POSIX AIO fsync yes 2023-10-31T23:29:39.622 INFO:teuthology.orchestra.run.smithi050.stdout:POSIX pshared support yes 2023-10-31T23:29:39.670 INFO:teuthology.orchestra.run.smithi050.stdout:pthread_condattr_setclock() yes 2023-10-31T23:29:39.708 INFO:teuthology.orchestra.run.smithi050.stdout:pthread_sigmask() yes 2023-10-31T23:29:39.744 INFO:teuthology.orchestra.run.smithi050.stdout:pthread_getaffinity_np() yes 2023-10-31T23:29:39.755 INFO:teuthology.orchestra.run.smithi050.stdout:Solaris AIO support no 2023-10-31T23:29:39.813 INFO:teuthology.orchestra.run.smithi050.stdout:__sync_fetch_and_add yes 2023-10-31T23:29:39.862 INFO:teuthology.orchestra.run.smithi050.stdout:__sync_synchronize yes 2023-10-31T23:29:39.892 INFO:teuthology.orchestra.run.smithi050.stdout:__sync_val_compare_and_swap yes 2023-10-31T23:29:39.899 INFO:teuthology.orchestra.run.smithi050.stdout:libverbs no 2023-10-31T23:29:39.928 INFO:teuthology.orchestra.run.smithi050.stdout:rdmacm no 2023-10-31T23:29:39.936 INFO:teuthology.orchestra.run.smithi050.stdout:librpma no 2023-10-31T23:29:39.946 INFO:teuthology.orchestra.run.smithi050.stdout:libprotobuf_c no 2023-10-31T23:29:39.977 INFO:teuthology.orchestra.run.smithi050.stdout:asprintf() yes 2023-10-31T23:29:40.008 INFO:teuthology.orchestra.run.smithi050.stdout:vasprintf() yes 2023-10-31T23:29:40.041 INFO:teuthology.orchestra.run.smithi050.stdout:Linux fallocate yes 2023-10-31T23:29:40.072 INFO:teuthology.orchestra.run.smithi050.stdout:POSIX fadvise yes 2023-10-31T23:29:40.104 INFO:teuthology.orchestra.run.smithi050.stdout:POSIX fallocate yes 2023-10-31T23:29:40.134 INFO:teuthology.orchestra.run.smithi050.stdout:sched_setaffinity(3 arg) yes 2023-10-31T23:29:40.135 INFO:teuthology.orchestra.run.smithi050.stdout:sched_setaffinity(2 arg) no 2023-10-31T23:29:40.166 INFO:teuthology.orchestra.run.smithi050.stdout:clock_gettime yes 2023-10-31T23:29:40.198 INFO:teuthology.orchestra.run.smithi050.stdout:CLOCK_MONOTONIC yes 2023-10-31T23:29:40.242 INFO:teuthology.orchestra.run.smithi050.stdout:clockid_t yes 2023-10-31T23:29:40.285 INFO:teuthology.orchestra.run.smithi050.stdout:gettimeofday yes 2023-10-31T23:29:40.317 INFO:teuthology.orchestra.run.smithi050.stdout:fdatasync yes 2023-10-31T23:29:40.349 INFO:teuthology.orchestra.run.smithi050.stdout:pipe() yes 2023-10-31T23:29:40.379 INFO:teuthology.orchestra.run.smithi050.stdout:pipe2() yes 2023-10-31T23:29:40.410 INFO:teuthology.orchestra.run.smithi050.stdout:pread() yes 2023-10-31T23:29:40.498 INFO:teuthology.orchestra.run.smithi050.stdout:sync_file_range yes 2023-10-31T23:29:40.505 INFO:teuthology.orchestra.run.smithi050.stdout:ASharedMemory_create no 2023-10-31T23:29:40.519 INFO:teuthology.orchestra.run.smithi050.stdout:EXT4 move extent yes 2023-10-31T23:29:40.551 INFO:teuthology.orchestra.run.smithi050.stdout:Linux splice(2) yes 2023-10-31T23:29:40.558 INFO:teuthology.orchestra.run.smithi050.stdout:libnuma no 2023-10-31T23:29:40.589 INFO:teuthology.orchestra.run.smithi050.stdout:strsep yes 2023-10-31T23:29:40.619 INFO:teuthology.orchestra.run.smithi050.stdout:strcasestr yes 2023-10-31T23:29:40.633 INFO:teuthology.orchestra.run.smithi050.stdout:strlcat no 2023-10-31T23:29:40.668 INFO:teuthology.orchestra.run.smithi050.stdout:getopt_long_only() yes 2023-10-31T23:29:40.735 INFO:teuthology.orchestra.run.smithi050.stdout:inet_aton yes 2023-10-31T23:29:40.767 INFO:teuthology.orchestra.run.smithi050.stdout:socklen_t yes 2023-10-31T23:29:40.800 INFO:teuthology.orchestra.run.smithi050.stdout:__thread yes 2023-10-31T23:29:40.831 INFO:teuthology.orchestra.run.smithi050.stdout:RUSAGE_THREAD yes 2023-10-31T23:29:40.861 INFO:teuthology.orchestra.run.smithi050.stdout:SCHED_IDLE yes 2023-10-31T23:29:40.895 INFO:teuthology.orchestra.run.smithi050.stdout:TCP_NODELAY yes 2023-10-31T23:29:40.930 INFO:teuthology.orchestra.run.smithi050.stdout:Net engine window_size yes 2023-10-31T23:29:40.965 INFO:teuthology.orchestra.run.smithi050.stdout:TCP_MAXSEG yes 2023-10-31T23:29:40.995 INFO:teuthology.orchestra.run.smithi050.stdout:RLIMIT_MEMLOCK yes 2023-10-31T23:29:41.029 INFO:teuthology.orchestra.run.smithi050.stdout:pwritev/preadv yes 2023-10-31T23:29:41.062 INFO:teuthology.orchestra.run.smithi050.stdout:pwritev2/preadv2 yes 2023-10-31T23:29:41.100 INFO:teuthology.orchestra.run.smithi050.stdout:IPv6 helpers yes 2023-10-31T23:29:41.116 INFO:teuthology.orchestra.run.smithi050.stdout:http engine no 2023-10-31T23:29:41.212 INFO:teuthology.orchestra.run.smithi050.stdout:Rados engine yes 2023-10-31T23:29:41.318 INFO:teuthology.orchestra.run.smithi050.stdout:Rados Block Device engine yes 2023-10-31T23:29:41.432 INFO:teuthology.orchestra.run.smithi050.stdout:rbd_poll yes 2023-10-31T23:29:41.542 INFO:teuthology.orchestra.run.smithi050.stdout:rbd_invalidate_cache yes 2023-10-31T23:29:41.573 INFO:teuthology.orchestra.run.smithi050.stdout:setvbuf yes 2023-10-31T23:29:41.580 INFO:teuthology.orchestra.run.smithi050.stdout:Gluster API engine no 2023-10-31T23:29:41.596 INFO:teuthology.orchestra.run.smithi050.stdout:s390_z196_facilities no 2023-10-31T23:29:41.596 INFO:teuthology.orchestra.run.smithi050.stdout:HDFS engine no 2023-10-31T23:29:41.642 INFO:teuthology.orchestra.run.smithi050.stdout:MTD yes 2023-10-31T23:29:41.649 INFO:teuthology.orchestra.run.smithi050.stdout:libpmem no 2023-10-31T23:29:41.649 INFO:teuthology.orchestra.run.smithi050.stdout:libpmem1_5 no 2023-10-31T23:29:41.657 INFO:teuthology.orchestra.run.smithi050.stdout:libpmem2 no 2023-10-31T23:29:41.657 INFO:teuthology.orchestra.run.smithi050.stdout:PMDK dev-dax engine no 2023-10-31T23:29:41.657 INFO:teuthology.orchestra.run.smithi050.stdout:PMDK libpmem engine no 2023-10-31T23:29:41.664 INFO:teuthology.orchestra.run.smithi050.stdout:DDN's Infinite Memory Engine no 2023-10-31T23:29:41.664 INFO:teuthology.orchestra.run.smithi050.stdout:iscsi engine no 2023-10-31T23:29:41.664 INFO:teuthology.orchestra.run.smithi050.stdout:NBD engine no 2023-10-31T23:29:41.673 INFO:teuthology.orchestra.run.smithi050.stdout:DAOS File System (dfs) Engine no 2023-10-31T23:29:41.682 INFO:teuthology.orchestra.run.smithi050.stdout:NFS engine no 2023-10-31T23:29:41.682 INFO:teuthology.orchestra.run.smithi050.stdout:lex/yacc for arithmetic no 2023-10-31T23:29:41.717 INFO:teuthology.orchestra.run.smithi050.stdout:getmntent yes 2023-10-31T23:29:41.740 INFO:teuthology.orchestra.run.smithi050.stdout:getmntinfo no 2023-10-31T23:29:41.798 INFO:teuthology.orchestra.run.smithi050.stdout:Static Assert yes 2023-10-31T23:29:41.825 INFO:teuthology.orchestra.run.smithi050.stdout:bool yes 2023-10-31T23:29:41.860 INFO:teuthology.orchestra.run.smithi050.stdout:strndup yes 2023-10-31T23:29:41.867 INFO:teuthology.orchestra.run.smithi050.stdout:Valgrind headers no 2023-10-31T23:29:41.914 INFO:teuthology.orchestra.run.smithi050.stdout:Zoned block device support yes 2023-10-31T23:29:41.942 INFO:teuthology.orchestra.run.smithi050.stdout:Zoned block device capacity yes 2023-10-31T23:29:41.949 INFO:teuthology.orchestra.run.smithi050.stdout:libzbc engine no 2023-10-31T23:29:41.983 INFO:teuthology.orchestra.run.smithi050.stdout:NVMe uring command support yes 2023-10-31T23:29:41.997 INFO:teuthology.orchestra.run.smithi050.stdout:xnvme engine no 2023-10-31T23:29:42.005 INFO:teuthology.orchestra.run.smithi050.stdout:isal no 2023-10-31T23:29:42.020 INFO:teuthology.orchestra.run.smithi050.stdout:libblkio engine no 2023-10-31T23:29:42.020 INFO:teuthology.orchestra.run.smithi050.stdout:march_armv8_a_crc_crypto no 2023-10-31T23:29:42.020 INFO:teuthology.orchestra.run.smithi050.stdout:cuda no 2023-10-31T23:29:42.020 INFO:teuthology.orchestra.run.smithi050.stdout:libcufile no 2023-10-31T23:29:42.048 INFO:teuthology.orchestra.run.smithi050.stdout:Build march=native yes 2023-10-31T23:29:42.055 INFO:teuthology.orchestra.run.smithi050.stdout:CUnit no 2023-10-31T23:29:42.084 INFO:teuthology.orchestra.run.smithi050.stdout:__kernel_rwf_t yes 2023-10-31T23:29:42.113 INFO:teuthology.orchestra.run.smithi050.stdout:-Wimplicit-fallthrough=2 yes 2023-10-31T23:29:42.144 INFO:teuthology.orchestra.run.smithi050.stdout:-Wno-stringop-truncation yes 2023-10-31T23:29:42.176 INFO:teuthology.orchestra.run.smithi050.stdout:MADV_HUGEPAGE yes 2023-10-31T23:29:42.208 INFO:teuthology.orchestra.run.smithi050.stdout:gettid yes 2023-10-31T23:29:42.242 INFO:teuthology.orchestra.run.smithi050.stdout:statx(2)/libc yes 2023-10-31T23:29:42.277 INFO:teuthology.orchestra.run.smithi050.stdout:statx(2)/syscall yes 2023-10-31T23:29:42.279 INFO:teuthology.orchestra.run.smithi050.stdout:Windows PDB generation no 2023-10-31T23:29:42.310 INFO:teuthology.orchestra.run.smithi050.stdout:timerfd_create yes 2023-10-31T23:29:42.312 INFO:teuthology.orchestra.run.smithi050.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2023-10-31T23:29:42.312 INFO:teuthology.orchestra.run.smithi050.stdout:Lib-based ioengines dynamic no 2023-10-31T23:29:42.376 INFO:teuthology.orchestra.run.smithi050.stdout:TCMalloc support yes 2023-10-31T23:29:42.377 INFO:teuthology.orchestra.run.smithi050.stdout:seed_buckets 4 2023-10-31T23:29:42.456 INFO:teuthology.orchestra.run.smithi050.stderr:FIO_VERSION = fio-3.36-11-g95f4 2023-10-31T23:29:42.491 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc16.o 2023-10-31T23:29:42.574 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc32.o 2023-10-31T23:29:42.610 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc32c-arm64.o 2023-10-31T23:29:42.691 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc32c-intel.o 2023-10-31T23:29:42.736 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc32c.o 2023-10-31T23:29:42.777 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc64.o 2023-10-31T23:29:42.816 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crc7.o 2023-10-31T23:29:42.850 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/crct10dif_common.o 2023-10-31T23:29:42.890 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/fnv.o 2023-10-31T23:29:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/md5.o 2023-10-31T23:29:43.035 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/murmur3.o 2023-10-31T23:29:43.077 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/sha1.o 2023-10-31T23:29:43.277 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/sha256.o 2023-10-31T23:29:43.583 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/sha3.o 2023-10-31T23:29:43.731 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/sha512.o 2023-10-31T23:29:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/test.o 2023-10-31T23:29:44.012 INFO:teuthology.orchestra.run.smithi050.stdout: CC crc/xxhash.o 2023-10-31T23:29:44.112 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/axmap.o 2023-10-31T23:29:44.239 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/bloom.o 2023-10-31T23:29:44.324 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/flist_sort.o 2023-10-31T23:29:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/gauss.o 2023-10-31T23:29:44.494 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/getrusage.o 2023-10-31T23:29:44.530 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/hweight.o 2023-10-31T23:29:44.568 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/ieee754.o 2023-10-31T23:29:44.618 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/lfsr.o 2023-10-31T23:29:44.720 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/memalign.o 2023-10-31T23:29:44.764 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/memcpy.o 2023-10-31T23:29:44.912 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/mountcheck.o 2023-10-31T23:29:44.954 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/num2str.o 2023-10-31T23:29:45.020 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/output_buffer.o 2023-10-31T23:29:45.067 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/pattern.o 2023-10-31T23:29:45.217 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/prio_tree.o 2023-10-31T23:29:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/rand.o 2023-10-31T23:29:45.534 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/rbtree.o 2023-10-31T23:29:45.663 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/strntol.o 2023-10-31T23:29:45.711 INFO:teuthology.orchestra.run.smithi050.stdout: CC lib/zipf.o 2023-10-31T23:29:45.836 INFO:teuthology.orchestra.run.smithi050.stdout: CC gettime.o 2023-10-31T23:29:46.137 INFO:teuthology.orchestra.run.smithi050.stdout: CC ioengines.o 2023-10-31T23:29:46.406 INFO:teuthology.orchestra.run.smithi050.stdout: CC init.o 2023-10-31T23:29:47.178 INFO:teuthology.orchestra.run.smithi050.stdout: CC stat.o 2023-10-31T23:29:49.113 INFO:teuthology.orchestra.run.smithi050.stdout: CC log.o 2023-10-31T23:29:49.241 INFO:teuthology.orchestra.run.smithi050.stdout: CC time.o 2023-10-31T23:29:49.365 INFO:teuthology.orchestra.run.smithi050.stdout: CC filesetup.o 2023-10-31T23:29:50.090 INFO:teuthology.orchestra.run.smithi050.stdout: CC eta.o 2023-10-31T23:29:50.416 INFO:teuthology.orchestra.run.smithi050.stdout: CC verify.o 2023-10-31T23:29:51.113 INFO:teuthology.orchestra.run.smithi050.stdout: CC memory.o 2023-10-31T23:29:51.296 INFO:teuthology.orchestra.run.smithi050.stdout: CC io_u.o 2023-10-31T23:29:52.084 INFO:teuthology.orchestra.run.smithi050.stdout: CC parse.o 2023-10-31T23:29:52.555 INFO:teuthology.orchestra.run.smithi050.stdout: CC fio_sem.o 2023-10-31T23:29:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: CC rwlock.o 2023-10-31T23:29:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: CC pshared.o 2023-10-31T23:29:52.815 INFO:teuthology.orchestra.run.smithi050.stdout: CC options.o 2023-10-31T23:29:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: CC smalloc.o 2023-10-31T23:29:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: CC filehash.o 2023-10-31T23:29:53.981 INFO:teuthology.orchestra.run.smithi050.stdout: CC profile.o 2023-10-31T23:29:54.102 INFO:teuthology.orchestra.run.smithi050.stdout: CC debug.o 2023-10-31T23:29:54.144 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/cpu.o 2023-10-31T23:29:54.283 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/mmap.o 2023-10-31T23:29:54.459 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/sync.o 2023-10-31T23:29:54.689 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/null.o 2023-10-31T23:29:54.807 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/net.o 2023-10-31T23:29:55.230 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/ftruncate.o 2023-10-31T23:29:55.332 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/fileoperations.o 2023-10-31T23:29:55.477 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/exec.o 2023-10-31T23:29:55.634 INFO:teuthology.orchestra.run.smithi050.stdout: CC server.o 2023-10-31T23:29:56.396 INFO:teuthology.orchestra.run.smithi050.stdout: CC client.o 2023-10-31T23:29:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: CC iolog.o 2023-10-31T23:29:57.412 INFO:teuthology.orchestra.run.smithi050.stdout: CC backend.o 2023-10-31T23:29:58.277 INFO:teuthology.orchestra.run.smithi050.stdout: CC libfio.o 2023-10-31T23:29:58.464 INFO:teuthology.orchestra.run.smithi050.stdout: CC flow.o 2023-10-31T23:29:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: CC cconv.o 2023-10-31T23:29:58.941 INFO:teuthology.orchestra.run.smithi050.stdout: CC gettime-thread.o 2023-10-31T23:29:59.047 INFO:teuthology.orchestra.run.smithi050.stdout: CC helpers.o 2023-10-31T23:29:59.109 INFO:teuthology.orchestra.run.smithi050.stdout: CC json.o 2023-10-31T23:29:59.310 INFO:teuthology.orchestra.run.smithi050.stdout: CC idletime.o 2023-10-31T23:29:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: CC td_error.o 2023-10-31T23:29:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: CC profiles/tiobench.o 2023-10-31T23:29:59.747 INFO:teuthology.orchestra.run.smithi050.stdout: CC profiles/act.o 2023-10-31T23:29:59.915 INFO:teuthology.orchestra.run.smithi050.stdout: CC io_u_queue.o 2023-10-31T23:29:59.965 INFO:teuthology.orchestra.run.smithi050.stdout: CC filelock.o 2023-10-31T23:30:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: CC workqueue.o 2023-10-31T23:30:00.272 INFO:teuthology.orchestra.run.smithi050.stdout: CC rate-submit.o 2023-10-31T23:30:00.424 INFO:teuthology.orchestra.run.smithi050.stdout: CC optgroup.o 2023-10-31T23:30:00.470 INFO:teuthology.orchestra.run.smithi050.stdout: CC helper_thread.o 2023-10-31T23:30:00.652 INFO:teuthology.orchestra.run.smithi050.stdout: CC steadystate.o 2023-10-31T23:30:00.894 INFO:teuthology.orchestra.run.smithi050.stdout: CC zone-dist.o 2023-10-31T23:30:01.029 INFO:teuthology.orchestra.run.smithi050.stdout: CC zbd.o 2023-10-31T23:30:01.746 INFO:teuthology.orchestra.run.smithi050.stdout: CC dedupe.o 2023-10-31T23:30:01.892 INFO:teuthology.orchestra.run.smithi050.stdout: CC fdp.o 2023-10-31T23:30:02.035 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/posixaio.o 2023-10-31T23:30:02.172 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/falloc.o 2023-10-31T23:30:02.289 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/e4defrag.o 2023-10-31T23:30:02.422 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/splice.o 2023-10-31T23:30:02.618 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/asprintf.o 2023-10-31T23:30:02.655 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/strlcat.o 2023-10-31T23:30:02.699 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/mtd.o 2023-10-31T23:30:02.868 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/libmtd.o 2023-10-31T23:30:03.254 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/libmtd_legacy.o 2023-10-31T23:30:03.389 INFO:teuthology.orchestra.run.smithi050.stdout: CC diskutil.o 2023-10-31T23:30:03.604 INFO:teuthology.orchestra.run.smithi050.stdout: CC fifo.o 2023-10-31T23:30:03.660 INFO:teuthology.orchestra.run.smithi050.stdout: CC blktrace.o 2023-10-31T23:30:03.998 INFO:teuthology.orchestra.run.smithi050.stdout: CC cgroup.o 2023-10-31T23:30:04.151 INFO:teuthology.orchestra.run.smithi050.stdout: CC trim.o 2023-10-31T23:30:04.267 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/sg.o 2023-10-31T23:30:04.661 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/linux-dev-lookup.o 2023-10-31T23:30:04.719 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/io_uring.o 2023-10-31T23:30:05.109 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/nvme.o 2023-10-31T23:30:05.404 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/linux-blkzoned.o 2023-10-31T23:30:05.560 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/cmdprio.o 2023-10-31T23:30:05.802 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/rados.o 2023-10-31T23:30:05.982 INFO:teuthology.orchestra.run.smithi050.stdout: CC engines/rbd.o 2023-10-31T23:30:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: CC fio.o 2023-10-31T23:30:06.264 INFO:teuthology.orchestra.run.smithi050.stdout: LINK fio 2023-10-31T23:30:06.471 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/log.o 2023-10-31T23:30:06.512 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/genzipf.o 2023-10-31T23:30:06.654 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/strcasestr.o 2023-10-31T23:30:06.678 INFO:teuthology.orchestra.run.smithi050.stdout: CC oslib/strndup.o 2023-10-31T23:30:06.701 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/fio-genzipf 2023-10-31T23:30:06.794 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/btrace2fio.o 2023-10-31T23:30:07.232 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/fio-btrace2fio 2023-10-31T23:30:07.331 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/verify-state.o 2023-10-31T23:30:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/debug.o 2023-10-31T23:30:07.466 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/fio-verify-state 2023-10-31T23:30:07.556 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/stest.o 2023-10-31T23:30:07.626 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/arch.o 2023-10-31T23:30:07.656 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/stest 2023-10-31T23:30:07.750 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/ieee754.o 2023-10-31T23:30:07.791 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/ieee754 2023-10-31T23:30:07.881 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/axmap.o 2023-10-31T23:30:07.982 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/axmap 2023-10-31T23:30:08.076 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/lfsr-test.o 2023-10-31T23:30:08.175 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/lfsr-test 2023-10-31T23:30:08.271 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/gen-rand.o 2023-10-31T23:30:08.370 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/gen-rand 2023-10-31T23:30:08.466 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/memlock.o 2023-10-31T23:30:08.527 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/memlock 2023-10-31T23:30:08.617 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/read-to-pipe-async.o 2023-10-31T23:30:08.808 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/read-to-pipe-async 2023-10-31T23:30:08.898 INFO:teuthology.orchestra.run.smithi050.stdout: CC t/io_uring.o 2023-10-31T23:30:09.458 INFO:teuthology.orchestra.run.smithi050.stdout: LINK t/io_uring 2023-10-31T23:30:09.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-10-31T23:30:09.583 DEBUG:teuthology.orchestra.run.smithi050:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-31T23:30:09.821 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - Settings.cluster: 2023-10-31T23:30:09.822 INFO:teuthology.orchestra.run.smithi050.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-10-31T23:30:09.822 INFO:teuthology.orchestra.run.smithi050.stderr: 'clients': ['ubuntu@smithi050.front.sepia.ceph.com'], 2023-10-31T23:30:09.822 INFO:teuthology.orchestra.run.smithi050.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-10-31T23:30:09.822 INFO:teuthology.orchestra.run.smithi050.stderr: 'head': 'ubuntu@smithi050.front.sepia.ceph.com', 2023-10-31T23:30:09.823 INFO:teuthology.orchestra.run.smithi050.stderr: 'is_teuthology': True, 2023-10-31T23:30:09.823 INFO:teuthology.orchestra.run.smithi050.stderr: 'iterations': 1, 2023-10-31T23:30:09.823 INFO:teuthology.orchestra.run.smithi050.stderr: 'mons': ['ubuntu@smithi050.front.sepia.ceph.com'], 2023-10-31T23:30:09.823 INFO:teuthology.orchestra.run.smithi050.stderr: 'osds': ['ubuntu@smithi050.front.sepia.ceph.com'], 2023-10-31T23:30:09.823 INFO:teuthology.orchestra.run.smithi050.stderr: 'osds_per_node': 3, 2023-10-31T23:30:09.824 INFO:teuthology.orchestra.run.smithi050.stderr: 'pid_dir': '/var/run/ceph', 2023-10-31T23:30:09.824 INFO:teuthology.orchestra.run.smithi050.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2023-10-31T23:30:09.824 INFO:teuthology.orchestra.run.smithi050.stderr: 'rebuild_every_test': False, 2023-10-31T23:30:09.824 INFO:teuthology.orchestra.run.smithi050.stderr: 'rgws': {'ubuntu@smithi050.front.sepia.ceph.com': None}, 2023-10-31T23:30:09.824 INFO:teuthology.orchestra.run.smithi050.stderr: 'tmp_dir': '/tmp/cbt', 2023-10-31T23:30:09.824 INFO:teuthology.orchestra.run.smithi050.stderr: 'use_existing': True, 2023-10-31T23:30:09.825 INFO:teuthology.orchestra.run.smithi050.stderr: 'user': 'ubuntu'} 2023-10-31T23:30:09.825 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.825 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw; fi 2023-10-31T23:30:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.827 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2023-10-31T23:30:09.829 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-10-31T23:30:09.829 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.829 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.830 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl 2023-10-31T23:30:09.832 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.832 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T23:30:09.834 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.834 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.834 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf 2023-10-31T23:30:09.836 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:30:09.837 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T23:30:09.837 INFO:teuthology.orchestra.run.smithi050.stderr:23:30:09 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T23:31:09.857 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.858 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T23:31:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.894 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T23:31:09.934 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.935 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* 2023-10-31T23:31:09.956 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.957 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/$(basename ${i}).smithi050.front.sepia.ceph.com ; done 2023-10-31T23:31:09.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl 2023-10-31T23:31:09.964 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.964 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T23:31:09.966 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.966 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.966 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf 2023-10-31T23:31:09.969 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.969 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T23:31:09.969 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T23:31:09.970 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.970 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:09.970 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2023-10-31T23:31:10.276 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:10 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:10.277 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:10 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:10.277 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2023-10-31T23:31:11.487 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:11 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:11.487 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:11 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:11.488 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2023-10-31T23:31:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:12 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:12 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:12.490 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:12 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2023-10-31T23:31:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:13 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:13 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:13.495 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2023-10-31T23:31:14.495 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:14 - INFO - cbt - Checking Health after pool creation. 2023-10-31T23:31:19.500 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:19 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:19.501 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:19 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:19.501 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-31T23:31:19.827 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:19 - INFO - cbt - Final Pool Health Check. 2023-10-31T23:31:24.832 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:24 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:24 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:24.833 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-31T23:31:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:25 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:25 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2023-10-31T23:31:25.153 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:25 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:25 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:25.154 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi050.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2023-10-31T23:31:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi050.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2023-10-31T23:31:26.910 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.911 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.911 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T23:31:26.928 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.928 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.928 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T23:31:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - INFO - cbt - Attempting to populating fio files... 2023-10-31T23:31:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-10-31T23:31:26.962 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:31:26.963 INFO:teuthology.orchestra.run.smithi050.stderr:23:31:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-10-31T23:33:59.305 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.306 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be. 2023-10-31T23:33:59.306 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - INFO - cbt - Setting OSD Read Ahead to: 4096 2023-10-31T23:33:59.306 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.306 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.306 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2023-10-31T23:33:59.307 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2023-10-31T23:33:59.308 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.308 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.308 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2023-10-31T23:33:59.326 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.326 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.326 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-31T23:33:59.873 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:33:59.873 INFO:teuthology.orchestra.run.smithi050.stderr:23:33:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-31T23:34:00.642 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:00 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:00.642 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:00 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:00.642 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2023-10-31T23:34:00.765 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:00 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:00.766 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:00 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:00.766 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/ceph_settings.out 2023-10-31T23:34:26.757 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:26 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:26.757 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:26 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:26.757 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log 2023-10-31T23:34:27.111 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.112 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.112 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/collectl 2023-10-31T23:34:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T23:34:27.115 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.116 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.116 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf 2023-10-31T23:34:27.119 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T23:34:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T23:34:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - INFO - cbt - Running rbd fio randrw test. 2023-10-31T23:34:27.121 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2023-10-31T23:34:27.121 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.122 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.122 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.0 2023-10-31T23:34:27.123 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2023-10-31T23:34:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:34:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:23:34:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/output.1 2023-10-31T23:35:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.031 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T23:35:28.053 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.053 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.053 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T23:35:28.087 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.087 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf.data 2023-10-31T23:35:28.088 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.088 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.* 2023-10-31T23:35:28.090 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.090 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.090 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/historic_ops.out 2023-10-31T23:35:28.395 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.410 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.411 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* 2023-10-31T23:35:28.430 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - get_fqdn_local()=smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.431 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/$(basename ${i}).smithi050.front.sepia.ceph.com ; done 2023-10-31T23:35:28.479 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - INFO - cbt - Convert results to json format. 2023-10-31T23:35:28.480 INFO:teuthology.orchestra.run.smithi050.stderr:23:35:28 - DEBUG - cbt - Nodes : ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.480 INFO:teuthology.orchestra.run.smithi050.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw 2023-10-31T23:35:28.480 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050.front.sepia.ceph.com: smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.481 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-10-31T23:35:28.481 INFO:teuthology.orchestra.run.smithi050.stdout:['smithi050.front.sepia.ceph.com'] 2023-10-31T23:35:28.498 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-10-31T23:35:28.508 DEBUG:teuthology.orchestra.run.smithi050:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-10-31T23:35:28.666 DEBUG:teuthology.orchestra.run.smithi050:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2023-10-31T23:35:28.726 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-10-31T23:35:28.726 INFO:tasks.cbt_performance:reading cbt preferences 2023-10-31T23:35:28.727 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-10-31T23:35:28.784 INFO:teuthology.orchestra.run.smithi050.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.0.smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.784 INFO:teuthology.orchestra.run.smithi050.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.1.smithi050.front.sepia.ceph.com 2023-10-31T23:35:28.785 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:35:28.786 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.0.smithi050.front.sepia.ceph.com of=/dev/stdout 2023-10-31T23:35:28.806 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T23:35:28.807 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be -name 'perf_stat.*' 2023-10-31T23:35:28.862 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T23:35:28.862 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T23:35:28.862 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-10-31T23:35:28.863 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be/json_output.1.smithi050.front.sepia.ceph.com of=/dev/stdout 2023-10-31T23:35:28.917 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T23:35:28.918 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-4c80a9be -name 'perf_stat.*' 2023-10-31T23:35:28.972 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T23:35:28.973 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T23:35:28.983 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T23:35:28.992 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T23:35:28.992 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-31T23:35:29.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-31T23:35:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:35:29.117+0000 7f729bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:35:29.125 INFO:teuthology.orchestra.run.smithi050.stderr:2023-10-31T23:35:29.125+0000 7f729bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T23:35:29.324 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-31T23:35:29.324 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-31T23:35:29.325 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-31T23:35:29.325 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T23:35:30.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-31T23:35:35.427 INFO:tasks.ceph.osd.0:Stopped 2023-10-31T23:35:35.428 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-31T23:35:35.428 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T23:35:36.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-31T23:35:36.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-31T23:35:41.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2023-10-31T23:35:41.530 INFO:tasks.ceph.osd.1:Stopped 2023-10-31T23:35:41.531 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-31T23:35:41.531 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T23:35:46.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-31T23:35:46.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-31T23:35:47.633 INFO:tasks.ceph.osd.2:Stopped 2023-10-31T23:35:47.634 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-31T23:35:47.634 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-31T23:35:47.635 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T23:35:47.665 INFO:tasks.ceph.mgr.x:Stopped 2023-10-31T23:35:47.665 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-31T23:35:47.666 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-31T23:35:47.666 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T23:35:47.718 INFO:tasks.ceph.mon.a:Stopped 2023-10-31T23:35:47.719 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-31T23:35:47.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-31T23:35:47.926 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:47.926 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-31T23:35:48.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:48.335 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-31T23:35:48.429 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:35:48.430 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-31T23:35:48.495 INFO:tasks.ceph:Archiving mon data... 2023-10-31T23:35:48.496 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530/data/mon.a.tgz 2023-10-31T23:35:48.497 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-10-31T23:35:48.520 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.RjuWy1Gxyo 2023-10-31T23:35:48.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.RjuWy1Gxyo 2023-10-31T23:35:48.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.RjuWy1Gxyo 2023-10-31T23:35:48.787 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.RjuWy1Gxyo is 207KB 2023-10-31T23:35:48.881 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.RjuWy1Gxyo 2023-10-31T23:35:48.896 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-31T23:35:48.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-31T23:35:48.978 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-31T23:35:48.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-31T23:35:49.172 INFO:teuthology.orchestra.run.smithi050.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-31T23:35:49.176 INFO:tasks.ceph:Archiving crash dumps... 2023-10-31T23:35:49.178 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530/remote/smithi050/crash 2023-10-31T23:35:49.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-31T23:35:49.209 INFO:tasks.ceph:Compressing logs... 2023-10-31T23:35:49.210 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-31T23:35:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80602.log 2023-10-31T23:35:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85685.log 2023-10-31T23:35:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-31T23:35:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.80602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81797.log 2023-10-31T23:35:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85685.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80602.log.gz 2023-10-31T23:35:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81799.log 2023-10-31T23:35:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81531.log 2023-10-31T23:35:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86210.log 2023-10-31T23:35:49.300 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81531.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-31T23:35:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85660.log 2023-10-31T23:35:49.301 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86210.log.gz 2023-10-31T23:35:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81038.log 2023-10-31T23:35:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85772.log 2023-10-31T23:35:49.301 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82096.log 2023-10-31T23:35:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85772.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-31T23:35:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.82096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85544.log 2023-10-31T23:35:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:gzip/var/log/ceph/ceph-osd.1.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81067.log 2023-10-31T23:35:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85804.log 2023-10-31T23:35:49.302 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82012.log 2023-10-31T23:35:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80840.log 2023-10-31T23:35:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.82012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85803.log 2023-10-31T23:35:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.80840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80800.log 2023-10-31T23:35:49.303 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81886.log 2023-10-31T23:35:49.303 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85685.log.gz 2023-10-31T23:35:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.80800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81573.log 2023-10-31T23:35:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82054.log 2023-10-31T23:35:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85708.log 2023-10-31T23:35:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.82054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86162.log 2023-10-31T23:35:49.304 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81800.log 2023-10-31T23:35:49.305 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81676.log 2023-10-31T23:35:49.305 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86162.log.gz 2023-10-31T23:35:49.305 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81800.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-31T23:35:49.305 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86498.log 2023-10-31T23:35:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85731.log 2023-10-31T23:35:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81475.log 2023-10-31T23:35:49.306 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86498.log.gz 2023-10-31T23:35:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80880.log 2023-10-31T23:35:49.306 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81475.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-31T23:35:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.80880.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-31T23:35:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81926.log 2023-10-31T23:35:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-31T23:35:49.307 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81925.log 2023-10-31T23:35:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85606.log 2023-10-31T23:35:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85755.log 2023-10-31T23:35:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80690.log 2023-10-31T23:35:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86209.log 2023-10-31T23:35:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.80690.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79243.log 2023-10-31T23:35:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.86209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81634.log 2023-10-31T23:35:49.309 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86209.log.gz 2023-10-31T23:35:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.tmp-client.admin.79243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85635.log 2023-10-31T23:35:49.309 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.81634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81718.log 2023-10-31T23:35:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.85635.log: /var/log/ceph/ceph-client.admin.81718.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80800.log.gz 2023-10-31T23:35:49.313 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.80840.log.gz 2023-10-31T23:35:49.313 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80880.log.gz 2023-10-31T23:35:49.313 INFO:teuthology.orchestra.run.smithi050.stderr: 66.4% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79243.log.gz 2023-10-31T23:35:49.314 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80690.log.gz 2023-10-31T23:35:49.315 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81038.log.gz 2023-10-31T23:35:49.317 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81067.log.gz 2023-10-31T23:35:49.325 INFO:teuthology.orchestra.run.smithi050.stderr: 85.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-31T23:35:49.331 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81475.log.gz 2023-10-31T23:35:49.331 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81531.log.gz 2023-10-31T23:35:49.332 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81573.log.gz 2023-10-31T23:35:49.332 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81634.log.gz 2023-10-31T23:35:49.332 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81676.log.gz 2023-10-31T23:35:49.332 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81718.log.gz 2023-10-31T23:35:49.333 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81799.log.gz 2023-10-31T23:35:49.333 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81797.log.gz 2023-10-31T23:35:49.333 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81800.log.gz 2023-10-31T23:35:49.333 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81886.log.gz 2023-10-31T23:35:49.333 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81926.log.gz 2023-10-31T23:35:49.333 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81925.log.gz 2023-10-31T23:35:49.334 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82012.log.gz 2023-10-31T23:35:49.334 INFO:teuthology.orchestra.run.smithi050.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.82054.log.gz 2023-10-31T23:35:49.334 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82096.log.gz 2023-10-31T23:35:49.342 INFO:teuthology.orchestra.run.smithi050.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-31T23:35:49.418 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85660.log.gz 2023-10-31T23:35:49.451 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85544.log.gz 2023-10-31T23:35:49.460 INFO:teuthology.orchestra.run.smithi050.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-31T23:35:49.467 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85606.log.gz 2023-10-31T23:35:49.468 INFO:teuthology.orchestra.run.smithi050.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85635.log.gz 2023-10-31T23:35:49.495 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85708.log.gz 2023-10-31T23:35:49.495 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85731.log.gz 2023-10-31T23:35:49.496 INFO:teuthology.orchestra.run.smithi050.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.85755.log.gz 2023-10-31T23:35:49.496 INFO:teuthology.orchestra.run.smithi050.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.85772.log.gz 2023-10-31T23:35:49.496 INFO:teuthology.orchestra.run.smithi050.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85803.log.gz 2023-10-31T23:35:49.496 INFO:teuthology.orchestra.run.smithi050.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85804.log.gz 2023-10-31T23:35:49.591 INFO:teuthology.orchestra.run.smithi050.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-31T23:35:51.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-31T23:36:07.092 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-31T23:36:07.368 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-31T23:36:07.439 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-31T23:36:07.441 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-10-31T23:36:07.441 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m18.175s 2023-10-31T23:36:07.441 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m11.581s 2023-10-31T23:36:07.442 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.589s 2023-10-31T23:36:07.442 INFO:tasks.ceph:Archiving logs... 2023-10-31T23:36:07.442 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530/remote/smithi050/log 2023-10-31T23:36:07.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-31T23:36:09.168 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-31T23:36:09.179 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-31T23:36:09.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-31T23:36:09.242 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-31T23:36:09.251 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-31T23:36:09.252 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-31T23:36:09.272 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2023-10-31T23:36:09.335 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-31T23:36:09.335 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-10-31T23:36:09.335 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +1096us[+1163us] +/- 76ms 2023-10-31T23:36:09.336 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -674us[ -607us] +/- 73ms 2023-10-31T23:36:09.336 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -306us[ -306us] +/- 75ms 2023-10-31T23:36:09.336 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T23:36:09.348 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-31T23:36:09.357 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-31T23:36:09.358 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-31T23:36:09.400 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-31T23:36:09.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-31T23:36:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T23:36:09.613 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-31T23:36:09.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-31T23:36:09.633 INFO:teuthology.task.internal:Duration was 958.749789 seconds 2023-10-31T23:36:09.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-31T23:36:09.668 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-31T23:36:09.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-31T23:36:09.782 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-31T23:36:10.238 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-31T23:36:10.238 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-10-31T23:36:10.239 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-31T23:36:10.260 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-31T23:36:10.261 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-31T23:36:10.334 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-31T23:36:10.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-31T23:36:10.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-31T23:36:10.486 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-31T23:36:10.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-31T23:36:10.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-31T23:36:10.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-31T23:36:10.595 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-10-31T23:36:10.624 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-31T23:36:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T23:36:10.639 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-31T23:36:10.650 INFO:teuthology.task.internal:Transferring archived files... 2023-10-31T23:36:10.650 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442530/remote/smithi050 2023-10-31T23:36:10.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-31T23:36:10.757 INFO:teuthology.task.internal:Removing archive directory... 2023-10-31T23:36:10.758 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-31T23:36:10.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-31T23:36:10.817 INFO:teuthology.task.internal:Not uploading archives. 2023-10-31T23:36:10.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-31T23:36:10.871 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-31T23:36:10.872 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-31T23:36:10.887 INFO:teuthology.orchestra.run.smithi050.stdout: 265629 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 31 23:36 /home/ubuntu/cephtest 2023-10-31T23:36:10.888 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-31T23:36:10.908 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-31T23:36:10.948 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-matanb-crimson-do_osd_ops_execute-part3&var-job_id_selected=7442530 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} duration: 958.7497889995575 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-10-31T23:36:10.948 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-31T23:36:11.033 INFO:teuthology.run:pass