2023-10-11T04:16:22.947 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-11T04:16:22.948 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057 --name yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} --owner scheduled_yingxin@teuthology -- /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057/orig.config.yaml 2023-10-11T04:16:22.995 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T04:16:23.077 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057 branch: wip-yingxin-crimson-osd-crosscore-pg-submission description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7421057' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-crimson-osd-crosscore-pg-submission ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 workunit: branch: wip-yingxin-crimson-osd-crosscore-pg-submission env: CRIMSON_COMPAT: '1' sha1: 9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-crimson-osd-crosscore-pg-submission suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPb/2XcsmNxT2spAc0MKWdISk0DgxsLAtY3d/1DcGJUHuy4/8FBEwiCUD6fl9rgDRJJugcWzy7LzSRkyJG1gGqs= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-11_02:17:04 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-11T04:16:23.078 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/qa; will attempt to use it 2023-10-11T04:16:23.078 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/qa/tasks 2023-10-11T04:16:23.079 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-11T04:16:23.081 INFO:teuthology.task.internal:Checking packages... 2023-10-11T04:16:23.103 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5' 2023-10-11T04:16:23.103 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-11T04:16:23.104 INFO:teuthology.packaging:ref: None 2023-10-11T04:16:23.104 INFO:teuthology.packaging:tag: None 2023-10-11T04:16:23.104 INFO:teuthology.packaging:branch: wip-yingxin-crimson-osd-crosscore-pg-submission 2023-10-11T04:16:23.105 INFO:teuthology.packaging:sha1: 9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 2023-10-11T04:16:23.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-crimson-osd-crosscore-pg-submission 2023-10-11T04:16:23.244 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6684.g9587ffdf 2023-10-11T04:16:23.244 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-11T04:16:23.252 INFO:teuthology.task.internal:no buildpackages task found 2023-10-11T04:16:23.252 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-11T04:16:23.266 INFO:teuthology.task.internal:Saving configuration 2023-10-11T04:16:23.282 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-11T04:16:23.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-11T04:16:23.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-11 04:11:29.429039', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPb/2XcsmNxT2spAc0MKWdISk0DgxsLAtY3d/1DcGJUHuy4/8FBEwiCUD6fl9rgDRJJugcWzy7LzSRkyJG1gGqs='} 2023-10-11T04:16:23.320 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-11T04:16:23.332 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-11T04:16:23.332 INFO:teuthology.run_tasks:Running task console_log... 2023-10-11T04:16:23.383 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc3fd383940>, signals=[15]) 2023-10-11T04:16:23.385 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-11T04:16:23.396 INFO:teuthology.task.internal:Opening connections... 2023-10-11T04:16:23.396 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:16:23.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T04:16:23.498 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-11T04:16:23.511 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-10-11T04:16:23.576 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-10-11T04:16:23.577 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-10-11T04:16:23.634 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2023-10-11T04:16:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2023-10-11T04:16:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2023-10-11T04:16:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2023-10-11T04:16:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2023-10-11T04:16:23.635 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2023-10-11T04:16:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-11T04:16:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-10-11T04:16:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2023-10-11T04:16:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-11T04:16:23.636 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2023-10-11T04:16:23.637 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-11T04:16:23.637 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-11T04:16:23.637 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-11T04:16:23.638 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-10-11T04:16:23.669 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-11T04:16:23.683 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-11T04:16:23.694 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-11T04:16:23.694 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-10-11T04:16:23.712 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-11T04:16:23.723 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-11T04:16:23.723 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-10-11T04:16:23.776 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-11T04:16:23.813 INFO:teuthology.run_tasks:Running task kernel... 2023-10-11T04:16:23.832 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-11T04:16:23.832 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-11T04:16:23.833 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-11T04:16:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T04:16:23.851 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-10-11T04:16:23.909 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-361.el9.x86_64 2023-10-11T04:16:23.910 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-361.el9.x86_64 2023-10-11T04:16:23.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2023-10-11T04:16:25.955 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-11T04:16:27.306 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-10-11T04:16:28.147 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-10-11T04:16:30.831 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-10-11T04:16:35.626 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.7 MB 00:01 2023-10-11T04:16:36.907 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 14 kB 00:00 2023-10-11T04:16:37.834 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 23 MB/s | 19 MB 00:00 2023-10-11T04:16:42.467 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2023-10-11T04:16:44.565 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-11T04:16:44.566 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-11T04:16:44.661 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:16:44.668 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:16:44.669 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:16:44.669 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:16:44.669 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:16:44.669 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-373.el9 CentOS-BaseOS 5.2 M 2023-10-11T04:16:44.670 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:16:44.670 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 5.14.0-373.el9 CentOS-BaseOS 20 M 2023-10-11T04:16:44.670 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 5.14.0-373.el9 CentOS-BaseOS 37 M 2023-10-11T04:16:44.670 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core x86_64 5.14.0-373.el9 CentOS-BaseOS 32 M 2023-10-11T04:16:44.670 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:16:44.671 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:16:44.671 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:16:44.671 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2023-10-11T04:16:44.671 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:16:44.672 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 94 M 2023-10-11T04:16:44.672 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 120 M 2023-10-11T04:16:44.672 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:16:44.985 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): kernel-5.14.0-373.el9.x86_64.rpm 17 MB/s | 5.2 MB 00:00 2023-10-11T04:16:45.535 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): kernel-core-5.14.0-373.el9.x86_64.rpm 23 MB/s | 20 MB 00:00 2023-10-11T04:16:46.069 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): kernel-modules-core-5.14.0-373.el9.x86_6 30 MB/s | 32 MB 00:01 2023-10-11T04:16:46.569 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): kernel-modules-5.14.0-373.el9.x86_64.rpm 20 MB/s | 37 MB 00:01 2023-10-11T04:16:46.572 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:16:46.573 INFO:teuthology.orchestra.run.smithi040.stdout:Total 50 MB/s | 94 MB 00:01 2023-10-11T04:16:46.573 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:16:47.011 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:16:47.012 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:16:48.025 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:16:48.026 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:16:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:16:51.211 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-core-5.14.0-373.el9.x86_64 1/4 2023-10-11T04:16:51.344 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-5.14.0-373.el9.x86_64 2/4 2023-10-11T04:16:54.226 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-373.el9.x86_64 2/4 2023-10-11T04:16:54.397 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-5.14.0-373.el9.x86_64 3/4 2023-10-11T04:16:59.368 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-373.el9.x86_64 3/4 2023-10-11T04:16:59.465 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-5.14.0-373.el9.x86_64 4/4 2023-10-11T04:17:04.449 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-core-5.14.0-373.el9.x86_64 4/4 2023-10-11T04:17:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-373.el9.x86_64 4/4 2023-10-11T04:17:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-373.el9.x86_64 4/4 2023-10-11T04:17:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-373.el9.x86_64 4/4 2023-10-11T04:17:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-373.el9.x86_64 1/4 2023-10-11T04:17:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-5.14.0-373.el9.x86_64 2/4 2023-10-11T04:17:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-5.14.0-373.el9.x86_64 3/4 2023-10-11T04:17:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-core-5.14.0-373.el9.x86_64 4/4 2023-10-11T04:17:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:17:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:17:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-373.el9.x86_64 2023-10-11T04:17:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core-5.14.0-373.el9.x86_64 2023-10-11T04:17:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-5.14.0-373.el9.x86_64 2023-10-11T04:17:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core-5.14.0-373.el9.x86_64 2023-10-11T04:17:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:17:36.132 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:17:36.632 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T04:17:36.764 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-373.el9.x86_64 2023-10-11T04:17:36.765 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-373.el9.x86_64 2023-10-11T04:17:36.765 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-373.el9.x86_64 2023-10-11T04:17:36.765 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-11T04:17:36.766 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-11T04:17:36.766 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-11T04:17:36.766 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2023-10-11T04:17:36.767 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-10-11T04:17:36.877 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-373.el9.x86_64 2023-10-11T04:17:36.877 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-373.el9.x86_64 2023-10-11T04:17:36.878 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2023-10-11T04:17:36.916 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2023-10-11T04:17:36.916 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2023-10-11T04:17:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2023-10-11T04:17:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2023-10-11T04:17:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2023-10-11T04:17:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-11T04:17:36.917 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2023-10-11T04:17:36.918 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2023-10-11T04:17:36.918 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2023-10-11T04:17:36.918 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-11T04:17:36.918 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-11T04:17:36.918 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-11T04:17:36.919 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-11T04:17:36.919 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2023-10-11T04:17:36.919 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2023-10-11T04:17:36.919 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-11T04:17:36.919 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2023-10-11T04:17:36.920 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2023-10-11T04:17:36.920 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:17:36.920 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-11T04:17:36.920 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-11T04:17:36.920 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-11T04:17:36.920 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2023-10-11T04:17:36.921 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:17:36.921 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2023-10-11T04:17:36.922 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-11T04:17:36.922 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 5.14.0-373.el9.x86_64 2023-10-11T04:17:36.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-11T04:17:37.595 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2023-10-11T04:17:39.016 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-11T04:17:39.053 INFO:teuthology.orchestra.run.smithi040.stderr:done 2023-10-11T04:17:39.055 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-10-11T04:17:39.110 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.jme3mlD9oA 2023-10-11T04:17:39.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jme3mlD9oA 2023-10-11T04:17:39.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.jme3mlD9oA 2023-10-11T04:17:39.327 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.jme3mlD9oA is 6KB 2023-10-11T04:17:39.376 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.jme3mlD9oA 2023-10-11T04:17:39.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2023-10-11T04:17:39.460 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-11T04:17:39.460 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-11T04:17:39.460 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-11T04:17:39.461 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-373.el9.x86_64.conf 2023-10-11T04:17:39.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-373.el9.x86_64 2023-10-11T04:17:39.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2023-10-11T04:17:39.717 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-10-11T04:18:09.647 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-11T04:18:09.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-11T04:18:09.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T04:19:09.710 DEBUG:teuthology.orchestra.remote:timed out 2023-10-11T04:19:18.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-11T04:19:18.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T04:19:18.986 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-10-11T04:19:19.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-11T04:19:19.555 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-373.el9.x86_64"... 2023-10-11T04:19:19.555 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-10-11T04:19:19.570 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-373.el9.x86_64 2023-10-11T04:19:19.570 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-373.el9.x86_64 vs 5.14.0-373.el9.x86_64 2023-10-11T04:19:19.570 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-11T04:19:19.571 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-11T04:19:20.572 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-11T04:19:20.572 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-11T04:19:20.654 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2023-10-11T04:19:20.706 DEBUG:teuthology.parallel:result is None 2023-10-11T04:19:20.707 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-11T04:19:20.714 INFO:teuthology.task.internal:Creating test directory... 2023-10-11T04:19:20.714 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-11T04:19:20.730 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-11T04:19:20.737 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-11T04:19:20.743 INFO:teuthology.task.internal:Creating archive directory... 2023-10-11T04:19:20.744 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-11T04:19:20.797 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-11T04:19:20.803 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-11T04:19:20.803 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-11T04:19:20.867 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T04:19:20.883 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-11T04:19:20.885 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-11T04:19:20.892 INFO:teuthology.task.internal:Configuring sudo... 2023-10-11T04:19:20.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-11T04:19:20.956 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-11T04:19:20.965 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-11T04:19:20.966 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-11T04:19:21.012 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T04:19:21.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-11T04:19:21.150 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T04:19:21.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-11T04:19:21.280 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:19:21.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-11T04:19:21.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-10-11T04:19:21.459 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T04:19:21.715 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-11T04:19:21.724 INFO:teuthology.task.internal:Starting timer... 2023-10-11T04:19:21.724 INFO:teuthology.run_tasks:Running task pcp... 2023-10-11T04:19:21.734 INFO:teuthology.run_tasks:Running task selinux... 2023-10-11T04:19:21.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2023-10-11T04:19:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2023-10-11T04:19:21.799 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-11T04:19:21.799 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2023-10-11T04:19:21.828 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2023-10-11T04:19:21.829 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive'} 2023-10-11T04:19:21.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T04:19:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T04:19:21.902 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-11T04:19:21.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2023-10-11T04:19:21.973 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-11T04:19:21.986 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-11T04:19:21.986 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-11T04:19:22.004 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-11T04:19:22.005 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-11T04:24:55.417 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com')] 2023-10-11T04:24:55.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-11T04:24:55.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-11T04:24:55.560 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-10-11T04:24:55.630 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-10-11T04:24:55.630 INFO:teuthology.run_tasks:Running task clock... 2023-10-11T04:24:55.645 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-11T04:24:55.646 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-11T04:24:55.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T04:24:55.708 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-11T04:24:55.728 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-11T04:24:55.766 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2023-10-11T04:24:55.783 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2023-10-11T04:24:55.804 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-11T04:24:55.824 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-11T04:24:55.871 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2023-10-11T04:24:55.874 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T04:24:55.875 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-10-11T04:24:55.875 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T04:24:55.875 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T04:24:55.875 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T04:24:55.875 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T04:24:55.876 INFO:teuthology.run_tasks:Running task install... 2023-10-11T04:24:55.884 DEBUG:teuthology.task.install:project ceph 2023-10-11T04:24:55.885 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5'}} 2023-10-11T04:24:55.885 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5'} 2023-10-11T04:24:55.885 INFO:teuthology.task.install:Using flavor: crimson 2023-10-11T04:24:55.891 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-11T04:24:55.892 INFO:teuthology.task.install:extra packages: [] 2023-10-11T04:24:55.892 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5', 'tag': None, 'wait_for_package': False} 2023-10-11T04:24:55.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 2023-10-11T04:24:56.118 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yingxin-crimson-osd-crosscore-pg-submission/9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/centos/9/flavors/crimson/ 2023-10-11T04:24:56.118 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6684.g9587ffdf 2023-10-11T04:24:56.288 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yingxin-crimson-osd-crosscore-pg-submission/9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yingxin-crimson-osd-crosscore-pg-submission/9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yingxin-crimson-osd-crosscore-pg-submission/9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-11T04:24:56.289 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:24:56.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-11T04:24:56.322 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-11T04:24:56.323 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-11T04:24:56.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-11T04:24:56.485 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-11T04:24:56.516 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2023-10-11T04:24:56.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2023-10-11T04:24:56.768 INFO:teuthology.orchestra.run.smithi040.stdout:70 files removed 2023-10-11T04:24:56.795 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2023-10-11T04:24:57.546 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 196 kB/s | 81 kB 00:00 2023-10-11T04:24:57.938 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-10-11T04:24:58.356 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 5.2 kB/s | 2.0 kB 00:00 2023-10-11T04:24:58.799 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-11T04:25:00.150 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-10-11T04:25:01.691 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 7.9 MB 00:01 2023-10-11T04:25:04.442 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-10-11T04:25:08.776 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 5.7 MB 00:00 2023-10-11T04:25:10.069 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 14 kB 00:00 2023-10-11T04:25:11.154 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 19 MB/s | 19 MB 00:00 2023-10-11T04:25:15.812 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-10-11T04:25:18.231 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:25:18.236 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:25:18.236 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:25:18.236 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:25:18.236 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:25:18.237 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 8.4 M 2023-10-11T04:25:18.237 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2023-10-11T04:25:18.237 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 3.1 M 2023-10-11T04:25:18.237 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 3.0 M 2023-10-11T04:25:18.238 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:25:18.238 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options 2023-10-11T04:25:18.239 INFO:teuthology.orchestra.run.smithi040.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-11T04:25:18.239 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 6.6 M 2023-10-11T04:25:18.239 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 17 M 2023-10-11T04:25:18.239 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 24 k 2023-10-11T04:25:18.239 INFO:teuthology.orchestra.run.smithi040.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-11T04:25:18.240 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-11T04:25:18.240 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-11T04:25:18.240 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-11T04:25:18.240 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 727 k 2023-10-11T04:25:18.240 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-11T04:25:18.241 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-11T04:25:18.241 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-11T04:25:18.241 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 469 k 2023-10-11T04:25:18.241 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-11T04:25:18.241 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 4.5 M 2023-10-11T04:25:18.242 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-11T04:25:18.242 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-11T04:25:18.242 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-11T04:25:18.242 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse 2023-10-11T04:25:18.242 INFO:teuthology.orchestra.run.smithi040.stdout: x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 45 k 2023-10-11T04:25:18.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 127 k 2023-10-11T04:25:18.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 162 k 2023-10-11T04:25:18.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-11T04:25:18.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 321 k 2023-10-11T04:25:18.243 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 298 k 2023-10-11T04:25:18.244 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 99 k 2023-10-11T04:25:18.244 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-11T04:25:18.244 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-11T04:25:18.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:18.244 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:25:18.245 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:25:18.245 INFO:teuthology.orchestra.run.smithi040.stdout:Install 28 Packages 2023-10-11T04:25:18.245 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2023-10-11T04:25:18.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:18.246 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 54 M 2023-10-11T04:25:18.246 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:25:19.288 INFO:teuthology.orchestra.run.smithi040.stdout:(1/30): ceph-base-18.0.0-6684.g9587ffdf.el9.x86 8.4 MB/s | 6.6 MB 00:00 2023-10-11T04:25:19.347 INFO:teuthology.orchestra.run.smithi040.stdout:(2/30): ceph-selinux-18.0.0-6684.g9587ffdf.el9. 411 kB/s | 24 kB 00:00 2023-10-11T04:25:19.539 INFO:teuthology.orchestra.run.smithi040.stdout:(3/30): ceph-radosgw-18.0.0-6684.g9587ffdf.el9. 8.1 MB/s | 8.4 MB 00:01 2023-10-11T04:25:19.564 INFO:teuthology.orchestra.run.smithi040.stdout:(4/30): libcephfs2-18.0.0-6684.g9587ffdf.el9.x8 3.3 MB/s | 727 kB 00:00 2023-10-11T04:25:19.606 INFO:teuthology.orchestra.run.smithi040.stdout:(5/30): libradosstriper1-18.0.0-6684.g9587ffdf. 6.8 MB/s | 469 kB 00:00 2023-10-11T04:25:19.681 INFO:teuthology.orchestra.run.smithi040.stdout:(6/30): python3-ceph-argparse-18.0.0-6684.g9587 604 kB/s | 45 kB 00:00 2023-10-11T04:25:19.748 INFO:teuthology.orchestra.run.smithi040.stdout:(7/30): python3-ceph-common-18.0.0-6684.g9587ff 1.9 MB/s | 127 kB 00:00 2023-10-11T04:25:19.824 INFO:teuthology.orchestra.run.smithi040.stdout:(8/30): python3-cephfs-18.0.0-6684.g9587ffdf.el 2.1 MB/s | 162 kB 00:00 2023-10-11T04:25:19.891 INFO:teuthology.orchestra.run.smithi040.stdout:(9/30): python3-rados-18.0.0-6684.g9587ffdf.el9 4.7 MB/s | 321 kB 00:00 2023-10-11T04:25:19.974 INFO:teuthology.orchestra.run.smithi040.stdout:(10/30): python3-rbd-18.0.0-6684.g9587ffdf.el9. 3.5 MB/s | 298 kB 00:00 2023-10-11T04:25:20.058 INFO:teuthology.orchestra.run.smithi040.stdout:(11/30): librgw2-18.0.0-6684.g9587ffdf.el9.x86_ 9.1 MB/s | 4.5 MB 00:00 2023-10-11T04:25:20.083 INFO:teuthology.orchestra.run.smithi040.stdout:(12/30): python3-rgw-18.0.0-6684.g9587ffdf.el9. 916 kB/s | 99 kB 00:00 2023-10-11T04:25:20.192 INFO:teuthology.orchestra.run.smithi040.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 792 kB/s | 106 kB 00:00 2023-10-11T04:25:20.225 INFO:teuthology.orchestra.run.smithi040.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-10-11T04:25:20.259 INFO:teuthology.orchestra.run.smithi040.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-10-11T04:25:20.284 INFO:teuthology.orchestra.run.smithi040.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-11T04:25:20.301 INFO:teuthology.orchestra.run.smithi040.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2023-10-11T04:25:20.393 INFO:teuthology.orchestra.run.smithi040.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2023-10-11T04:25:20.693 INFO:teuthology.orchestra.run.smithi040.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 283 kB/s | 111 kB 00:00 2023-10-11T04:25:20.835 INFO:teuthology.orchestra.run.smithi040.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2023-10-11T04:25:21.011 INFO:teuthology.orchestra.run.smithi040.stdout:(21/30): ceph-common-18.0.0-6684.g9587ffdf.el9. 6.9 MB/s | 17 MB 00:02 2023-10-11T04:25:21.027 INFO:teuthology.orchestra.run.smithi040.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 127 kB/s | 24 kB 00:00 2023-10-11T04:25:21.120 INFO:teuthology.orchestra.run.smithi040.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2023-10-11T04:25:21.295 INFO:teuthology.orchestra.run.smithi040.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 4.7 MB/s | 837 kB 00:00 2023-10-11T04:25:21.320 INFO:teuthology.orchestra.run.smithi040.stdout:(25/30): liboath-2.6.7-2.el9.x86_64.rpm 157 kB/s | 49 kB 00:00 2023-10-11T04:25:21.395 INFO:teuthology.orchestra.run.smithi040.stdout:(26/30): libarrow-9.0.0-9.el9.x86_64.rpm 6.3 MB/s | 4.4 MB 00:00 2023-10-11T04:25:21.421 INFO:teuthology.orchestra.run.smithi040.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2023-10-11T04:25:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2023-10-11T04:25:22.221 INFO:teuthology.orchestra.run.smithi040.stdout:(29/30): librbd1-18.0.0-6684.g9587ffdf.el9.x86_ 3.8 MB/s | 3.0 MB 00:00 2023-10-11T04:25:22.480 INFO:teuthology.orchestra.run.smithi040.stdout:(30/30): librados2-18.0.0-6684.g9587ffdf.el9.x8 2.9 MB/s | 3.1 MB 00:01 2023-10-11T04:25:22.484 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:25:22.484 INFO:teuthology.orchestra.run.smithi040.stdout:Total 13 MB/s | 54 MB 00:04 2023-10-11T04:25:22.496 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:25:22.539 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:25:22.540 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:25:23.080 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:25:23.081 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:25:24.473 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:25:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-11T04:25:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/32 2023-10-11T04:25:25.017 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/32 2023-10-11T04:25:25.107 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:18.0.0-6684.g9587ffdf.el9.x86_64 3/32 2023-10-11T04:25:25.141 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:18.0.0-6684.g9587ffdf.el9.x86_64 4/32 2023-10-11T04:25:25.196 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:18.0.0-6684.g9587ffdf.el9.x86_64 4/32 2023-10-11T04:25:25.242 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-11T04:25:25.271 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-11T04:25:25.309 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-11T04:25:25.353 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:18.0.0-6684.g9587ffdf.el9. 8/32 2023-10-11T04:25:25.431 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:18.0.0-6684.g9587ffdf.el9.x86_64 9/32 2023-10-11T04:25:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:18.0.0-6684.g9587ffdf.el9.x86_6 10/32 2023-10-11T04:25:25.575 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6684.g9587ffdf.el9.x86_6 10/32 2023-10-11T04:25:25.621 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-11T04:25:25.729 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-11T04:25:25.762 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-11T04:25:25.810 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-11T04:25:25.835 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-11T04:25:25.853 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-11T04:25:25.889 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-11T04:25:25.933 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-11T04:25:26.250 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-11T04:25:26.277 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:18.0.0-6684.g9587ffdf.el9.x86_64 20/32 2023-10-11T04:25:26.360 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:18.0.0-6684.g9587ffdf.el9.x86_64 20/32 2023-10-11T04:25:26.390 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 21/32 2023-10-11T04:25:26.426 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-11T04:25:26.881 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-11T04:25:26.904 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:18.0.0-6684.g9587ffdf.el9.x86_64 24/32 2023-10-11T04:25:26.959 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:18.0.0-6684.g9587ffdf.el9.x86_64 24/32 2023-10-11T04:25:27.038 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 25/32 2023-10-11T04:25:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:18.0.0-6684.g9587ffdf.el9.x8 26/32 2023-10-11T04:25:29.256 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-6684.g9587ffdf.el9.x86_64 27/32 2023-10-11T04:25:29.294 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:18.0.0-6684.g9587ffdf.el9.x86_64 27/32 2023-10-11T04:25:30.082 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:18.0.0-6684.g9587ffdf.el9.x86_64 27/32 2023-10-11T04:25:30.122 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:18.0.0-6684.g9587ffdf.el9.x86_64 28/32 2023-10-11T04:25:30.213 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:18.0.0-6684.g9587ffdf.el9.x86_64 28/32 2023-10-11T04:25:30.214 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-11T04:25:30.214 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-11T04:25:30.214 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:30.248 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:18.0.0-6684.g9587ffdf.el9.x86_64 29/32 2023-10-11T04:25:40.466 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6684.g9587ffdf.el9.x86_64 29/32 2023-10-11T04:25:40.467 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2023-10-11T04:25:40.467 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2023-10-11T04:25:40.467 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2023-10-11T04:25:40.468 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2023-10-11T04:25:40.468 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2023-10-11T04:25:40.468 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2023-10-11T04:25:40.469 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2023-10-11T04:25:40.469 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:41.358 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 30/32 2023-10-11T04:25:41.385 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 30/32 2023-10-11T04:25:41.385 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:25:41.386 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-11T04:25:41.386 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-11T04:25:41.386 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-11T04:25:41.387 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:41.388 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-11T04:25:41.470 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-11T04:25:41.470 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-11T04:25:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-11T04:25:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/32 2023-10-11T04:25:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/32 2023-10-11T04:25:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 3/32 2023-10-11T04:25:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:18.0.0-6684.g9587ffdf.el9.x86_64 4/32 2023-10-11T04:25:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:18.0.0-6684.g9587ffdf.el9.x86_64 5/32 2023-10-11T04:25:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:18.0.0-6684.g9587ffdf.el9.x86_6 6/32 2023-10-11T04:25:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:18.0.0-6684.g9587ffdf.el9.x86_64 7/32 2023-10-11T04:25:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6684.g9587ffdf.el9. 8/32 2023-10-11T04:25:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:18.0.0-6684.g9587ffdf.el9.x8 9/32 2023-10-11T04:25:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:18.0.0-6684.g9587ffdf.el9.x86_64 10/32 2023-10-11T04:25:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:18.0.0-6684.g9587ffdf.el9.x86_64 11/32 2023-10-11T04:25:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 12/32 2023-10-11T04:25:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 13/32 2023-10-11T04:25:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-11T04:25:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-11T04:25:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-11T04:25:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-11T04:25:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-11T04:25:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-11T04:25:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-11T04:25:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-11T04:25:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-11T04:25:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-11T04:25:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-11T04:25:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-11T04:25:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-11T04:25:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-11T04:25:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-11T04:25:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:18.0.0-6684.g9587ffdf.el9.x86_64 29/32 2023-10-11T04:25:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-11T04:25:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:18.0.0-6684.g9587ffdf.el9.x86_64 31/32 2023-10-11T04:25:43.636 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-11T04:25:43.636 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:43.636 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2023-10-11T04:25:43.636 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.637 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.637 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:25:43.637 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-11T04:25:43.637 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.637 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.638 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.638 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.638 INFO:teuthology.orchestra.run.smithi040.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-11T04:25:43.638 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-11T04:25:43.638 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-11T04:25:43.639 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-11T04:25:43.639 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.639 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-11T04:25:43.639 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-11T04:25:43.639 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-11T04:25:43.640 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.640 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-11T04:25:43.640 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.640 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-11T04:25:43.640 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-11T04:25:43.641 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-11T04:25:43.641 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.641 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.641 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.641 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-11T04:25:43.642 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.642 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.642 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:25:43.642 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2023-10-11T04:25:43.642 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-11T04:25:43.642 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:43.643 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:25:43.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2023-10-11T04:25:44.602 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:29 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:25:44.752 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:25:44.753 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:25:44.754 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:25:44.754 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:25:44.754 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:25:44.754 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 117 M 2023-10-11T04:25:44.754 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:25:44.755 INFO:teuthology.orchestra.run.smithi040.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-11T04:25:44.755 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-11T04:25:44.755 INFO:teuthology.orchestra.run.smithi040.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-11T04:25:44.755 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 161 k 2023-10-11T04:25:44.755 INFO:teuthology.orchestra.run.smithi040.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-11T04:25:44.756 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-11T04:25:44.756 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-11T04:25:44.756 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-11T04:25:44.756 INFO:teuthology.orchestra.run.smithi040.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-11T04:25:44.756 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:44.757 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:25:44.757 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:25:44.757 INFO:teuthology.orchestra.run.smithi040.stdout:Install 10 Packages 2023-10-11T04:25:44.757 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:25:44.758 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 120 M 2023-10-11T04:25:44.758 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.1 G 2023-10-11T04:25:44.758 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:25:45.143 INFO:teuthology.orchestra.run.smithi040.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.4 MB/s | 418 kB 00:00 2023-10-11T04:25:45.185 INFO:teuthology.orchestra.run.smithi040.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-10-11T04:25:45.226 INFO:teuthology.orchestra.run.smithi040.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-10-11T04:25:45.268 INFO:teuthology.orchestra.run.smithi040.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-10-11T04:25:45.294 INFO:teuthology.orchestra.run.smithi040.stdout:(5/10): libcephsqlite-18.0.0-6684.g9587ffdf.el9 503 kB/s | 161 kB 00:00 2023-10-11T04:25:45.321 INFO:teuthology.orchestra.run.smithi040.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-10-11T04:25:45.461 INFO:teuthology.orchestra.run.smithi040.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-10-11T04:25:45.545 INFO:teuthology.orchestra.run.smithi040.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 5.6 MB/s | 1.3 MB 00:00 2023-10-11T04:25:45.595 INFO:teuthology.orchestra.run.smithi040.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 913 kB/s | 122 kB 00:00 2023-10-11T04:25:50.945 INFO:teuthology.orchestra.run.smithi040.stdout:(10/10): ceph-test-18.0.0-6684.g9587ffdf.el9.x8 20 MB/s | 117 MB 00:05 2023-10-11T04:25:50.947 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:25:50.948 INFO:teuthology.orchestra.run.smithi040.stdout:Total 19 MB/s | 120 MB 00:06 2023-10-11T04:25:50.962 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:25:50.987 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:25:50.987 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:25:51.866 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:25:51.867 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:25:53.004 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:25:53.098 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-11T04:25:53.145 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-11T04:25:53.189 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-11T04:25:53.230 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-11T04:25:53.263 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-11T04:25:53.294 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-11T04:25:53.329 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-11T04:25:53.364 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-11T04:25:53.391 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:18.0.0-6684.g9587ffdf.el9.x86_64 9/10 2023-10-11T04:26:10.656 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6684.g9587ffdf.el9.x86_64 9/10 2023-10-11T04:26:10.732 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:18.0.0-6684.g9587ffdf.el9.x86_64 10/10 2023-10-11T04:26:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:18.0.0-6684.g9587ffdf.el9.x86_64 10/10 2023-10-11T04:26:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/10 2023-10-11T04:26:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/10 2023-10-11T04:26:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-11T04:26:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-11T04:26:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-11T04:26:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-11T04:26:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-11T04:26:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-11T04:26:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-11T04:26:19.509 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-11T04:26:19.509 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:19.509 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:26:19.509 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:26:19.509 INFO:teuthology.orchestra.run.smithi040.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-11T04:26:19.510 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-15.el9.x86_64 2023-10-11T04:26:19.510 INFO:teuthology.orchestra.run.smithi040.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-11T04:26:19.510 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:26:19.510 INFO:teuthology.orchestra.run.smithi040.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-11T04:26:19.510 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-11T04:26:19.511 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-11T04:26:19.511 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-11T04:26:19.511 INFO:teuthology.orchestra.run.smithi040.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-11T04:26:19.511 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:19.511 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:26:19.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2023-10-11T04:26:20.421 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:05 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:26:20.568 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:26:20.571 INFO:teuthology.orchestra.run.smithi040.stdout:========================================================================================= 2023-10-11T04:26:20.571 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:26:20.571 INFO:teuthology.orchestra.run.smithi040.stdout:========================================================================================= 2023-10-11T04:26:20.572 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:26:20.572 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 6.5 k 2023-10-11T04:26:20.572 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:26:20.572 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 2.0 M 2023-10-11T04:26:20.572 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 1.5 M 2023-10-11T04:26:20.573 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 242 k 2023-10-11T04:26:20.573 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 8.8 M 2023-10-11T04:26:20.573 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 117 M 2023-10-11T04:26:20.573 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-11T04:26:20.573 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-11T04:26:20.574 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-11T04:26:20.574 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-11T04:26:20.574 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-11T04:26:20.574 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-11T04:26:20.575 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-11T04:26:20.575 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-11T04:26:20.575 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-11T04:26:20.575 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-11T04:26:20.576 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-11T04:26:20.576 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-11T04:26:20.576 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-11T04:26:20.576 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-11T04:26:20.576 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-11T04:26:20.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-11T04:26:20.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-11T04:26:20.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-11T04:26:20.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-11T04:26:20.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-11T04:26:20.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-11T04:26:20.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-11T04:26:20.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-11T04:26:20.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-11T04:26:20.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-11T04:26:20.579 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-11T04:26:20.580 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-11T04:26:20.580 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:20.580 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:26:20.580 INFO:teuthology.orchestra.run.smithi040.stdout:========================================================================================= 2023-10-11T04:26:20.580 INFO:teuthology.orchestra.run.smithi040.stdout:Install 33 Packages 2023-10-11T04:26:20.581 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:20.581 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 134 M 2023-10-11T04:26:20.582 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.4 G 2023-10-11T04:26:20.582 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:26:21.181 INFO:teuthology.orchestra.run.smithi040.stdout:(1/33): ceph-18.0.0-6684.g9587ffdf.el9.x86_64.r 33 kB/s | 6.5 kB 00:00 2023-10-11T04:26:21.506 INFO:teuthology.orchestra.run.smithi040.stdout:(2/33): ceph-mgr-18.0.0-6684.g9587ffdf.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2023-10-11T04:26:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:(3/33): ceph-mds-18.0.0-6684.g9587ffdf.el9.x86_ 3.4 MB/s | 2.0 MB 00:00 2023-10-11T04:26:21.649 INFO:teuthology.orchestra.run.smithi040.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6684.g9587 2.8 MB/s | 242 kB 00:00 2023-10-11T04:26:21.749 INFO:teuthology.orchestra.run.smithi040.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-10-11T04:26:21.774 INFO:teuthology.orchestra.run.smithi040.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-10-11T04:26:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-10-11T04:26:21.841 INFO:teuthology.orchestra.run.smithi040.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-11T04:26:21.875 INFO:teuthology.orchestra.run.smithi040.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-11T04:26:21.909 INFO:teuthology.orchestra.run.smithi040.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-10-11T04:26:21.942 INFO:teuthology.orchestra.run.smithi040.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-11T04:26:21.976 INFO:teuthology.orchestra.run.smithi040.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 5.5 MB/s | 188 kB 00:00 2023-10-11T04:26:22.159 INFO:teuthology.orchestra.run.smithi040.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 6.7 MB/s | 1.2 MB 00:00 2023-10-11T04:26:22.252 INFO:teuthology.orchestra.run.smithi040.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 557 kB/s | 51 kB 00:00 2023-10-11T04:26:22.368 INFO:teuthology.orchestra.run.smithi040.stdout:(15/33): ceph-mon-18.0.0-6684.g9587ffdf.el9.x86 7.4 MB/s | 8.8 MB 00:01 2023-10-11T04:26:22.544 INFO:teuthology.orchestra.run.smithi040.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 149 kB/s | 43 kB 00:00 2023-10-11T04:26:22.752 INFO:teuthology.orchestra.run.smithi040.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 448 kB/s | 172 kB 00:00 2023-10-11T04:26:22.777 INFO:teuthology.orchestra.run.smithi040.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2023-10-11T04:26:22.894 INFO:teuthology.orchestra.run.smithi040.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2023-10-11T04:26:22.911 INFO:teuthology.orchestra.run.smithi040.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2023-10-11T04:26:22.995 INFO:teuthology.orchestra.run.smithi040.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2023-10-11T04:26:23.054 INFO:teuthology.orchestra.run.smithi040.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 137 kB/s | 19 kB 00:00 2023-10-11T04:26:23.112 INFO:teuthology.orchestra.run.smithi040.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2023-10-11T04:26:23.279 INFO:teuthology.orchestra.run.smithi040.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 473 kB/s | 79 kB 00:00 2023-10-11T04:26:23.321 INFO:teuthology.orchestra.run.smithi040.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 173 kB/s | 46 kB 00:00 2023-10-11T04:26:23.446 INFO:teuthology.orchestra.run.smithi040.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2023-10-11T04:26:23.596 INFO:teuthology.orchestra.run.smithi040.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 601 kB/s | 90 kB 00:00 2023-10-11T04:26:23.630 INFO:teuthology.orchestra.run.smithi040.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 777 kB/s | 272 kB 00:00 2023-10-11T04:26:23.772 INFO:teuthology.orchestra.run.smithi040.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 204 kB/s | 36 kB 00:00 2023-10-11T04:26:23.956 INFO:teuthology.orchestra.run.smithi040.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2023-10-11T04:26:23.981 INFO:teuthology.orchestra.run.smithi040.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 655 kB/s | 230 kB 00:00 2023-10-11T04:26:24.165 INFO:teuthology.orchestra.run.smithi040.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 96 kB/s | 20 kB 00:00 2023-10-11T04:26:28.315 INFO:teuthology.orchestra.run.smithi040.stdout:(33/33): ceph-osd-18.0.0-6684.g9587ffdf.el9.x86 17 MB/s | 117 MB 00:06 2023-10-11T04:26:28.319 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:26:28.319 INFO:teuthology.orchestra.run.smithi040.stdout:Total 17 MB/s | 134 MB 00:07 2023-10-11T04:26:28.353 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:26:28.397 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:26:28.397 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:26:29.427 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:26:29.428 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:26:31.316 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:26:31.376 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-11T04:26:31.408 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-11T04:26:31.441 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-11T04:26:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-11T04:26:31.614 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-11T04:26:31.661 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-11T04:26:31.698 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-11T04:26:31.762 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-11T04:26:31.812 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-10-11T04:26:31.850 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-11T04:26:31.909 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-11T04:26:31.972 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-11T04:26:32.178 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-11T04:26:32.230 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-11T04:26:32.268 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-11T04:26:32.303 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-11T04:26:32.332 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-11T04:26:32.412 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-11T04:26:32.442 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-11T04:26:32.476 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-11T04:26:32.509 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-11T04:26:32.535 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-11T04:26:32.625 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-11T04:26:32.668 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-11T04:26:32.743 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-11T04:26:32.886 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-11T04:26:33.020 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-11T04:26:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6684.g9587ffdf.el9. 28/33 2023-10-11T04:26:33.230 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:18.0.0-6684.g9587ffdf.el9.x86_64 29/33 2023-10-11T04:26:33.257 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6684.g9587ffdf.el9.x86_64 29/33 2023-10-11T04:26:33.258 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:26:33.258 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-11T04:26:33.258 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-11T04:26:33.259 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-11T04:26:33.259 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:52.680 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:18.0.0-6684.g9587ffdf.el9.x86_64 30/33 2023-10-11T04:26:52.707 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:18.0.0-6684.g9587ffdf.el9.x86_64 30/33 2023-10-11T04:26:52.708 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:26:52.708 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-11T04:26:52.708 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-11T04:26:52.708 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-11T04:26:52.708 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:53.647 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:18.0.0-6684.g9587ffdf.el9.x86_64 31/33 2023-10-11T04:26:53.673 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:18.0.0-6684.g9587ffdf.el9.x86_64 31/33 2023-10-11T04:26:53.673 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:26:53.674 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-11T04:26:53.674 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-11T04:26:53.674 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-11T04:26:53.674 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:53.881 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:18.0.0-6684.g9587ffdf.el9.x86_64 32/33 2023-10-11T04:26:53.906 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:18.0.0-6684.g9587ffdf.el9.x86_64 32/33 2023-10-11T04:26:53.907 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:26:53.907 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-11T04:26:53.907 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-11T04:26:53.907 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-11T04:26:53.907 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:26:53.972 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:18.0.0-6684.g9587ffdf.el9.x86_64 33/33 2023-10-11T04:27:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:18.0.0-6684.g9587ffdf.el9.x86_64 33/33 2023-10-11T04:27:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/33 2023-10-11T04:27:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/33 2023-10-11T04:27:03.845 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:18.0.0-6684.g9587ffdf.el9.x86_64 3/33 2023-10-11T04:27:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:18.0.0-6684.g9587ffdf.el9.x86_64 4/33 2023-10-11T04:27:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:18.0.0-6684.g9587ffdf.el9.x86_64 5/33 2023-10-11T04:27:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6684.g9587ffdf.el9. 6/33 2023-10-11T04:27:03.846 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-11T04:27:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-11T04:27:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-11T04:27:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-11T04:27:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-11T04:27:03.847 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-11T04:27:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-11T04:27:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-10-11T04:27:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-11T04:27:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-11T04:27:03.848 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-11T04:27:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-11T04:27:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-11T04:27:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-11T04:27:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-11T04:27:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-11T04:27:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-11T04:27:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-11T04:27:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-11T04:27:03.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-11T04:27:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-11T04:27:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-11T04:27:03.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-11T04:27:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-11T04:27:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-11T04:27:03.852 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-11T04:27:04.724 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-11T04:27:04.724 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:04.725 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:04.725 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:04.725 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:04.725 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:04.725 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:04.726 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:04.726 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:04.726 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-11T04:27:04.726 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-11T04:27:04.726 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-11T04:27:04.727 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-11T04:27:04.727 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-11T04:27:04.727 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-11T04:27:04.727 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-11T04:27:04.727 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-11T04:27:04.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-11T04:27:04.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-11T04:27:04.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-11T04:27:04.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-11T04:27:04.728 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-11T04:27:04.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-11T04:27:04.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-11T04:27:04.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2023-10-11T04:27:04.729 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-11T04:27:04.730 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-11T04:27:04.730 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-11T04:27:04.730 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-11T04:27:04.731 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-11T04:27:04.731 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-11T04:27:04.731 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-11T04:27:04.731 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-11T04:27:04.732 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-11T04:27:04.732 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-11T04:27:04.732 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-11T04:27:04.732 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:04.733 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:04.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2023-10-11T04:27:05.565 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:05.657 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:05.708 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:05.709 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:05.709 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:05.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install cephadm 2023-10-11T04:27:06.392 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:06.536 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:06.537 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:06.537 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:06.537 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:06.538 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:06.538 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 269 k 2023-10-11T04:27:06.538 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:06.538 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:06.538 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:06.539 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:27:06.539 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:06.539 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 269 k 2023-10-11T04:27:06.539 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 264 k 2023-10-11T04:27:06.539 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:06.980 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm-18.0.0-6684.g9587ffdf.el9.noarch.rpm 608 kB/s | 269 kB 00:00 2023-10-11T04:27:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:06.981 INFO:teuthology.orchestra.run.smithi040.stdout:Total 605 kB/s | 269 kB 00:00 2023-10-11T04:27:06.982 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:06.987 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:06.990 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:06.991 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:07.067 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:07.462 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:07.545 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:07.907 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:08.203 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:08.203 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:08.204 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:08.204 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:08.204 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:08.204 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:08.402 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache 2023-10-11T04:27:09.021 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:09.165 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:09.165 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:09.166 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-11T04:27:09.166 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:09.166 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:09.166 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 144 k 2023-10-11T04:27:09.167 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:09.167 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:09.167 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:09.167 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:27:09.167 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:09.168 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 144 k 2023-10-11T04:27:09.168 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 438 k 2023-10-11T04:27:09.168 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:09.703 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-18.0.0-6684.g9587ff 269 kB/s | 144 kB 00:00 2023-10-11T04:27:09.703 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:09.704 INFO:teuthology.orchestra.run.smithi040.stdout:Total 269 kB/s | 144 kB 00:00 2023-10-11T04:27:09.704 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:09.711 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:09.712 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:09.769 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:09.769 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:09.922 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:10.020 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6684.g9587ffdf. 1/1 2023-10-11T04:27:10.046 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6684.g9587ffdf. 1/1 2023-10-11T04:27:10.046 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:27:10.046 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-11T04:27:10.047 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:11.028 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6684.g9587ffdf. 1/1 2023-10-11T04:27:11.028 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:11.029 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:11.029 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:11.029 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:11.030 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:11.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2023-10-11T04:27:11.885 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:11.977 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:12.030 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:12.031 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:12.031 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:12.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2023-10-11T04:27:12.713 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:57 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:12.858 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:12.859 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:12.859 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:12.859 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:12.860 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:12.860 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 1.8 M 2023-10-11T04:27:12.860 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:27:12.860 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 23 k 2023-10-11T04:27:12.861 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 15 k 2023-10-11T04:27:12.861 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-11T04:27:12.861 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-11T04:27:12.861 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-11T04:27:12.862 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2023-10-11T04:27:12.862 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-11T04:27:12.862 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:12.862 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:12.863 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:12.863 INFO:teuthology.orchestra.run.smithi040.stdout:Install 7 Packages 2023-10-11T04:27:12.863 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:12.864 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.1 M 2023-10-11T04:27:12.864 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 8.6 M 2023-10-11T04:27:12.864 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:13.548 INFO:teuthology.orchestra.run.smithi040.stdout:(1/7): ceph-prometheus-alerts-18.0.0-6684.g9587 44 kB/s | 15 kB 00:00 2023-10-11T04:27:13.765 INFO:teuthology.orchestra.run.smithi040.stdout:(2/7): ceph-grafana-dashboards-18.0.0-6684.g958 42 kB/s | 23 kB 00:00 2023-10-11T04:27:13.782 INFO:teuthology.orchestra.run.smithi040.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 39 kB/s | 9.0 kB 00:00 2023-10-11T04:27:13.923 INFO:teuthology.orchestra.run.smithi040.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 258 kB/s | 41 kB 00:00 2023-10-11T04:27:14.015 INFO:teuthology.orchestra.run.smithi040.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 132 kB/s | 31 kB 00:00 2023-10-11T04:27:14.157 INFO:teuthology.orchestra.run.smithi040.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 807 kB/s | 188 kB 00:00 2023-10-11T04:27:14.741 INFO:teuthology.orchestra.run.smithi040.stdout:(7/7): ceph-mgr-dashboard-18.0.0-6684.g9587ffdf 1.2 MB/s | 1.8 MB 00:01 2023-10-11T04:27:14.743 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:14.743 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.1 MB/s | 2.1 MB 00:01 2023-10-11T04:27:14.744 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:14.754 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:14.754 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:14.834 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:14.835 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:15.209 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-11T04:27:15.244 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-11T04:27:15.290 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-11T04:27:15.321 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-11T04:27:15.350 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6684.g9587ffdf.el9.n 5/7 2023-10-11T04:27:15.597 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6684.g9587ffdf.el9. 6/7 2023-10-11T04:27:15.628 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6684.g9587ffdf.el9.noarc 7/7 2023-10-11T04:27:16.186 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6684.g9587ffdf.el9.noarc 7/7 2023-10-11T04:27:16.187 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6684.g9587ffdf.el9. 1/7 2023-10-11T04:27:16.187 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6684.g9587ffdf.el9.noarc 2/7 2023-10-11T04:27:16.187 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6684.g9587ffdf.el9.n 3/7 2023-10-11T04:27:16.187 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-11T04:27:16.187 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-11T04:27:16.188 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-11T04:27:16.496 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-11T04:27:16.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:16.496 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:16.496 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:16.496 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:16.497 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:16.497 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-11T04:27:16.497 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-11T04:27:16.498 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-11T04:27:16.498 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-11T04:27:16.498 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:16.499 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:16.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-11T04:27:17.319 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:02 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:17.465 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:17.466 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2023-10-11T04:27:17.467 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:17.467 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2023-10-11T04:27:17.467 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:17.467 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 7.4 M 2023-10-11T04:27:17.468 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:27:17.468 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-11T04:27:17.468 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-11T04:27:17.468 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-11T04:27:17.468 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-11T04:27:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-11T04:27:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-11T04:27:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-11T04:27:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-11T04:27:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-11T04:27:17.469 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-11T04:27:17.470 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-11T04:27:17.470 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2023-10-11T04:27:17.470 INFO:teuthology.orchestra.run.smithi040.stdout:Install 12 Packages 2023-10-11T04:27:17.471 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:17.471 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 37 M 2023-10-11T04:27:17.471 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 209 M 2023-10-11T04:27:17.472 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:17.589 INFO:teuthology.orchestra.run.smithi040.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 264 kB/s | 32 kB 00:00 2023-10-11T04:27:17.648 INFO:teuthology.orchestra.run.smithi040.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 503 kB/s | 17 kB 00:00 2023-10-11T04:27:17.681 INFO:teuthology.orchestra.run.smithi040.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-10-11T04:27:17.740 INFO:teuthology.orchestra.run.smithi040.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-10-11T04:27:17.833 INFO:teuthology.orchestra.run.smithi040.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 205 kB 00:00 2023-10-11T04:27:17.891 INFO:teuthology.orchestra.run.smithi040.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-10-11T04:27:17.983 INFO:teuthology.orchestra.run.smithi040.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-10-11T04:27:18.000 INFO:teuthology.orchestra.run.smithi040.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-10-11T04:27:18.258 INFO:teuthology.orchestra.run.smithi040.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2023-10-11T04:27:18.417 INFO:teuthology.orchestra.run.smithi040.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2023-10-11T04:27:18.443 INFO:teuthology.orchestra.run.smithi040.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-10-11T04:27:23.201 INFO:teuthology.orchestra.run.smithi040.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 1.3 MB/s | 7.4 MB 00:05 2023-10-11T04:27:23.204 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:23.204 INFO:teuthology.orchestra.run.smithi040.stdout:Total 6.5 MB/s | 37 MB 00:05 2023-10-11T04:27:23.204 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:23.230 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:23.231 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:23.695 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:23.696 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:24.450 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:24.529 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-11T04:27:24.563 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-11T04:27:24.664 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-11T04:27:24.717 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-11T04:27:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-11T04:27:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-11T04:27:25.023 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-11T04:27:25.738 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-11T04:27:25.859 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-11T04:27:27.208 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-11T04:27:28.295 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-11T04:27:28.323 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6684.g9587f 12/12 2023-10-11T04:27:30.867 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6684.g9587f 12/12 2023-10-11T04:27:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6684.g9587f 1/12 2023-10-11T04:27:30.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-11T04:27:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-11T04:27:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-11T04:27:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-11T04:27:30.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-11T04:27:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-11T04:27:30.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-11T04:27:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-11T04:27:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-11T04:27:30.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-11T04:27:31.280 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-11T04:27:31.280 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:31.280 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:31.280 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:31.280 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-11T04:27:31.281 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-11T04:27:31.281 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-11T04:27:31.281 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-11T04:27:31.281 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-11T04:27:31.281 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-11T04:27:31.282 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-11T04:27:31.282 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-11T04:27:31.282 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-11T04:27:31.282 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-11T04:27:31.283 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-11T04:27:31.283 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:31.283 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:31.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2023-10-11T04:27:32.290 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:17 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:32.436 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:32.437 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-11T04:27:32.437 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:32.437 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-11T04:27:32.438 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:32.438 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 47 k 2023-10-11T04:27:32.438 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:27:32.438 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-11T04:27:32.439 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-11T04:27:32.439 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-11T04:27:32.439 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-11T04:27:32.439 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-11T04:27:32.439 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-11T04:27:32.439 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-11T04:27:32.440 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-11T04:27:32.440 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-11T04:27:32.440 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-11T04:27:32.440 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-11T04:27:32.440 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-11T04:27:32.441 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:32.441 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:32.441 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-11T04:27:32.441 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2023-10-11T04:27:32.441 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:32.442 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.0 M 2023-10-11T04:27:32.442 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 26 M 2023-10-11T04:27:32.443 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:32.940 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2023-10-11T04:27:32.965 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2023-10-11T04:27:33.032 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2023-10-11T04:27:33.057 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2023-10-11T04:27:33.082 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-11T04:27:33.108 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-10-11T04:27:33.142 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): ceph-mgr-rook-18.0.0-6684.g9587ffdf.el9 142 kB/s | 47 kB 00:00 2023-10-11T04:27:33.341 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2023-10-11T04:27:33.375 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2023-10-11T04:27:33.517 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2023-10-11T04:27:33.559 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): python3-google-auth-2.23.2-1.el9.noarc 471 kB/s | 196 kB 00:00 2023-10-11T04:27:33.634 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 769 kB/s | 90 kB 00:00 2023-10-11T04:27:33.676 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2023-10-11T04:27:33.680 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:33.681 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-10-11T04:27:33.681 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:33.696 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:33.696 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:33.805 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:33.807 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:34.344 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:34.524 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-11T04:27:34.588 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-11T04:27:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-11T04:27:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-11T04:27:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-11T04:27:34.817 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-11T04:27:34.927 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 7/13 2023-10-11T04:27:34.981 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-11T04:27:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-11T04:27:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-11T04:27:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-11T04:27:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-11T04:27:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:18.0.0-6684.g9587ffdf.el9.noarch 13/13 2023-10-11T04:27:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6684.g9587ffdf.el9.noarch 13/13 2023-10-11T04:27:36.671 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6684.g9587ffdf.el9.noarch 1/13 2023-10-11T04:27:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-11T04:27:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-11T04:27:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-11T04:27:36.672 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-11T04:27:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-11T04:27:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-11T04:27:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-11T04:27:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-11T04:27:36.673 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 10/13 2023-10-11T04:27:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-11T04:27:36.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-11T04:27:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-11T04:27:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:37.044 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:37.044 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-11T04:27:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-11T04:27:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-11T04:27:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-11T04:27:37.045 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-11T04:27:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-11T04:27:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-11T04:27:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-11T04:27:37.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-11T04:27:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-11T04:27:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-11T04:27:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-11T04:27:37.047 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:37.047 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:37.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-cephadm 2023-10-11T04:27:37.878 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:22 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:38.022 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:38.023 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-11T04:27:38.023 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:38.024 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-11T04:27:38.024 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:38.024 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 132 k 2023-10-11T04:27:38.024 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:27:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-11T04:27:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-11T04:27:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-11T04:27:38.025 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-11T04:27:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-11T04:27:38.026 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:38.026 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:38.026 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-10-11T04:27:38.027 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2023-10-11T04:27:38.027 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:38.028 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 6.8 M 2023-10-11T04:27:38.028 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 33 M 2023-10-11T04:27:38.028 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:38.278 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-10-11T04:27:38.445 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-10-11T04:27:38.662 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6684.g9587ffdf.e 240 kB/s | 132 kB 00:00 2023-10-11T04:27:38.754 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 188 kB/s | 58 kB 00:00 2023-10-11T04:27:38.804 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2023-10-11T04:27:38.971 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2023-10-11T04:27:38.976 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:38.976 INFO:teuthology.orchestra.run.smithi040.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2023-10-11T04:27:38.976 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:38.987 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:38.987 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:39.084 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:39.085 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:39.388 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:39.554 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-11T04:27:39.607 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-11T04:27:40.169 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-11T04:27:40.254 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-11T04:27:40.321 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-11T04:27:40.352 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 6/6 2023-10-11T04:27:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 6/6 2023-10-11T04:27:41.181 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 1/6 2023-10-11T04:27:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-11T04:27:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-11T04:27:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-11T04:27:41.182 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-11T04:27:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-11T04:27:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:41.500 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-11T04:27:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-11T04:27:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-11T04:27:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-11T04:27:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-11T04:27:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:41.502 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:41.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2023-10-11T04:27:42.346 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:27 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:42.490 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:42.491 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:42.491 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:42.491 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:42.491 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:42.492 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 832 k 2023-10-11T04:27:42.492 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:27:42.492 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-11T04:27:42.492 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:42.492 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:42.493 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:42.493 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2023-10-11T04:27:42.493 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:42.493 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 911 k 2023-10-11T04:27:42.493 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 3.0 M 2023-10-11T04:27:42.494 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:42.615 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 640 kB/s | 79 kB 00:00 2023-10-11T04:27:42.799 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): ceph-fuse-18.0.0-6684.g9587ffdf.el9.x86_ 2.7 MB/s | 832 kB 00:00 2023-10-11T04:27:42.800 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:42.800 INFO:teuthology.orchestra.run.smithi040.stdout:Total 2.9 MB/s | 911 kB 00:00 2023-10-11T04:27:42.800 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:42.821 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:42.821 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:42.901 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:42.901 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:43.094 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:43.236 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-11T04:27:43.309 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/2 2023-10-11T04:27:43.994 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/2 2023-10-11T04:27:43.994 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/2 2023-10-11T04:27:44.314 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-11T04:27:44.315 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:44.315 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:44.315 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-11T04:27:44.315 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:44.316 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:44.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-volume 2023-10-11T04:27:45.192 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:30 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:45.337 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:45.338 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:45.339 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:27:45.339 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:45.339 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:45.339 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:18.0.0-6684.g9587ffdf.el9 ceph-noarch 262 k 2023-10-11T04:27:45.340 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:45.340 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:45.340 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:45.340 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:27:45.340 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:45.341 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 262 k 2023-10-11T04:27:45.341 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.2 M 2023-10-11T04:27:45.341 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:45.563 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-volume-18.0.0-6684.g9587ffdf.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-11T04:27:45.564 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:45.564 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-11T04:27:45.565 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:45.573 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:45.573 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:45.608 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:45.609 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:45.835 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:45.907 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:45.923 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:27:45.924 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-11T04:27:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:18.0.0-6684.g9587ffdf.el9.noarch 1/1 2023-10-11T04:27:46.989 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:46.990 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:18.0.0-6684.g9587ffdf.el9.noarch 2023-10-11T04:27:46.990 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:46.990 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:47.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2023-10-11T04:27:47.898 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:32 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:48.044 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:48.045 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:48.045 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-11T04:27:48.045 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:48.046 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:48.046 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 127 k 2023-10-11T04:27:48.046 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:48.046 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:48.046 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:27:48.047 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 127 k 2023-10-11T04:27:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 460 k 2023-10-11T04:27:48.047 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:48.251 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-18.0.0-6684.g9587ffdf.el9.x86_64 617 kB/s | 127 kB 00:00 2023-10-11T04:27:48.252 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:48.252 INFO:teuthology.orchestra.run.smithi040.stdout:Total 613 kB/s | 127 kB 00:00 2023-10-11T04:27:48.252 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:48.258 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:48.259 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:48.322 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:48.322 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:48.485 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:48.587 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:49.057 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:49.348 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:49.348 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:49.348 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:49.349 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:49.349 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:49.349 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:49.546 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2023-10-11T04:27:50.173 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:35 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:50.265 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:50.319 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:50.320 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:50.320 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:50.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2023-10-11T04:27:51.007 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:36 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:51.151 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:51.152 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:51.153 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-11T04:27:51.153 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:51.153 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:51.153 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 31 k 2023-10-11T04:27:51.153 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:51.154 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:51.154 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:51.154 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:27:51.154 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:51.154 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 31 k 2023-10-11T04:27:51.155 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 139 k 2023-10-11T04:27:51.155 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:51.397 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-18.0.0-6684.g9587ffdf.el9.x86_6 128 kB/s | 31 kB 00:00 2023-10-11T04:27:51.397 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:51.398 INFO:teuthology.orchestra.run.smithi040.stdout:Total 127 kB/s | 31 kB 00:00 2023-10-11T04:27:51.398 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:51.401 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:51.401 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:51.438 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:51.438 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:51.559 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:51.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:52.058 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:52.322 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:52.322 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:52.322 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:52.322 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:52.323 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:52.323 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:52.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2023-10-11T04:27:53.135 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:38 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:53.227 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:53.280 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:53.281 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:53.282 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:53.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2023-10-11T04:27:53.957 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:38 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:54.049 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:54.102 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:54.104 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:54.104 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:54.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados 2023-10-11T04:27:54.788 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:39 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:54.880 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:54.935 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:54.936 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:54.936 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:55.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw 2023-10-11T04:27:55.621 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:40 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:55.713 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:55.766 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:55.767 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:55.767 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:55.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs 2023-10-11T04:27:56.452 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:41 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:56.544 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:56.596 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:56.597 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:56.598 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:56.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd 2023-10-11T04:27:57.290 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:42 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:57.382 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 is already installed. 2023-10-11T04:27:57.435 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:57.435 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-10-11T04:27:57.435 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:57.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2023-10-11T04:27:58.116 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:43 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:27:58.262 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:27:58.263 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:58.263 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-10-11T04:27:58.263 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:58.263 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:27:58.264 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 87 k 2023-10-11T04:27:58.264 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:58.264 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:27:58.264 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:27:58.264 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:27:58.265 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:58.265 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 87 k 2023-10-11T04:27:58.265 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 252 k 2023-10-11T04:27:58.265 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:27:58.550 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-18.0.0-6684.g9587ffdf.el9.x86_64.rpm 303 kB/s | 87 kB 00:00 2023-10-11T04:27:58.551 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:27:58.551 INFO:teuthology.orchestra.run.smithi040.stdout:Total 301 kB/s | 87 kB 00:00 2023-10-11T04:27:58.552 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:27:58.559 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:27:58.559 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:27:58.617 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:27:58.617 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:27:58.756 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:27:58.850 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:59.232 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:59.503 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:27:59.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:59.588 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:27:59.588 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:27:59.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:27:59.588 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:27:59.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2023-10-11T04:28:00.337 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:45 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:28:00.483 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:28:00.483 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:00.484 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-10-11T04:28:00.484 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:00.484 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:28:00.484 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 3.0 M 2023-10-11T04:28:00.485 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:00.485 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:28:00.485 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:00.485 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:28:00.485 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:00.486 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.0 M 2023-10-11T04:28:00.486 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 13 M 2023-10-11T04:28:00.486 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:28:01.471 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-18.0.0-6684.g9587ffdf.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2023-10-11T04:28:01.471 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:28:01.472 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.0 MB/s | 3.0 MB 00:00 2023-10-11T04:28:01.472 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:28:01.481 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:28:01.481 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:28:01.558 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:28:01.559 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:28:02.007 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:28:02.071 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:28:02.097 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:28:02.097 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-11T04:28:02.098 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-11T04:28:02.098 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-11T04:28:02.098 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-11T04:28:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:03.245 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:28:03.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:03.245 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:28:03.246 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:28:03.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:03.246 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:28:03.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2023-10-11T04:28:04.100 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:49 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:28:04.245 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:28:04.246 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-10-11T04:28:04.247 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:04.247 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:28:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 168 k 2023-10-11T04:28:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:04.247 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:28:04.248 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:04.248 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-10-11T04:28:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:04.248 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 168 k 2023-10-11T04:28:04.249 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 533 k 2023-10-11T04:28:04.249 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:28:04.791 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-18.0.0-6684.g9587ffdf.el9.x86_64.rpm 310 kB/s | 168 kB 00:00 2023-10-11T04:28:04.791 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:28:04.792 INFO:teuthology.orchestra.run.smithi040.stdout:Total 309 kB/s | 168 kB 00:00 2023-10-11T04:28:04.792 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:28:04.800 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:28:04.801 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:28:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:28:04.866 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:28:05.031 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:28:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:28:05.608 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:28:05.946 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/1 2023-10-11T04:28:05.947 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:05.947 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:28:05.947 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:28:05.947 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:05.948 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:28:06.177 DEBUG:teuthology.parallel:result is None 2023-10-11T04:28:06.178 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5 2023-10-11T04:28:06.450 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-11T04:28:06.472 INFO:teuthology.orchestra.run.smithi040.stdout:18.0.0-6684.g9587ffdf.el9 2023-10-11T04:28:06.472 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6684.g9587ffdf.el9 2023-10-11T04:28:06.473 INFO:teuthology.task.install:The correct ceph version 18.0.0-6684.g9587ffdf is installed. 2023-10-11T04:28:06.473 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-11T04:28:06.474 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:06.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-11T04:28:06.544 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-11T04:28:06.545 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:06.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-10-11T04:28:06.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-11T04:28:06.688 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-11T04:28:06.688 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:06.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-11T04:28:06.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-11T04:28:06.828 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-11T04:28:06.829 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:06.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2023-10-11T04:28:06.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-11T04:28:06.968 INFO:teuthology.run_tasks:Running task ceph... 2023-10-11T04:28:07.052 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-11T04:28:07.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2023-10-11T04:28:07.085 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-11T04:28:07.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-11T04:28:07.155 INFO:tasks.ceph:Creating extra log directories... 2023-10-11T04:28:07.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-11T04:28:07.236 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-11T04:28:07.236 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-11T04:28:07.237 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057', 'branch': 'wip-yingxin-crimson-osd-crosscore-pg-submission', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7421057', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-crimson-osd-crosscore-pg-submission'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5'}}, 'workunit': {'branch': 'wip-yingxin-crimson-osd-crosscore-pg-submission', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-crimson-osd-crosscore-pg-submission', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5', 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPb/2XcsmNxT2spAc0MKWdISk0DgxsLAtY3d/1DcGJUHuy4/8FBEwiCUD6fl9rgDRJJugcWzy7LzSRkyJG1gGqs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9587ffdfc8c52bcf06abbe89d2f527c434a3e9f5', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-11_02:17:04', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-11T04:28:07.238 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-11T04:28:07.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2023-10-11T04:28:07.352 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:07.352 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2023-10-11T04:28:07.408 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-11T04:28:07.408 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2023-10-11T04:28:07.462 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-11T04:28:07.463 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T04:28:07.463 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-11T04:28:07.463 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T04:28:07.463 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T04:28:07.464 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-11 04:28:02.331223824 +0000 2023-10-11T04:28:07.464 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-11 04:24:30.745190709 +0000 2023-10-11T04:28:07.464 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-11 04:24:30.745190709 +0000 2023-10-11T04:28:07.464 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-11 04:24:30.745190709 +0000 2023-10-11T04:28:07.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-11T04:28:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-11T04:28:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-11T04:28:07.531 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000198169 s, 2.6 MB/s 2023-10-11T04:28:07.532 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-11T04:28:07.591 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2023-10-11T04:28:07.646 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-11T04:28:07.647 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T04:28:07.647 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-10-11T04:28:07.647 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T04:28:07.647 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T04:28:07.648 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-11 04:28:02.332223811 +0000 2023-10-11T04:28:07.648 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-11 04:24:31.258183970 +0000 2023-10-11T04:28:07.648 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-11 04:24:31.258183970 +0000 2023-10-11T04:28:07.648 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-11 04:24:31.258183970 +0000 2023-10-11T04:28:07.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-11T04:28:07.715 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-11T04:28:07.715 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-11T04:28:07.716 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000197948 s, 2.6 MB/s 2023-10-11T04:28:07.716 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-11T04:28:07.772 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2023-10-11T04:28:07.827 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-11T04:28:07.828 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T04:28:07.828 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-11T04:28:07.828 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T04:28:07.828 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T04:28:07.828 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-11 04:28:02.332223811 +0000 2023-10-11T04:28:07.829 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-11 04:24:31.773177204 +0000 2023-10-11T04:28:07.829 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-11 04:24:31.773177204 +0000 2023-10-11T04:28:07.829 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-11 04:24:31.773177204 +0000 2023-10-11T04:28:07.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-11T04:28:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-11T04:28:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-11T04:28:07.898 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000182218 s, 2.8 MB/s 2023-10-11T04:28:07.899 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-11T04:28:07.954 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2023-10-11T04:28:08.009 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-11T04:28:08.010 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-11T04:28:08.010 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-10-11T04:28:08.010 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-11T04:28:08.010 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-10-11T04:28:08.011 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-10-11 04:28:02.333223798 +0000 2023-10-11T04:28:08.011 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-10-11 04:24:32.295170347 +0000 2023-10-11T04:28:08.011 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-10-11 04:24:32.295170347 +0000 2023-10-11T04:28:08.011 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2023-10-11 04:24:32.295170347 +0000 2023-10-11T04:28:08.012 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-11T04:28:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-10-11T04:28:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-10-11T04:28:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000202317 s, 2.5 MB/s 2023-10-11T04:28:08.081 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-11T04:28:08.138 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T04:28:08.138 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T04:28:08.138 INFO:tasks.ceph:Generating config... 2023-10-11T04:28:08.140 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-11T04:28:08.140 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-11T04:28:08.140 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-10-11T04:28:08.141 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-11T04:28:08.142 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-11T04:28:08.142 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-10-11T04:28:08.145 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-11T04:28:08.145 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-11T04:28:08.145 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-11T04:28:08.146 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-10-11T04:28:08.147 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-10-11T04:28:08.147 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-10-11T04:28:08.147 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-10-11T04:28:08.148 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-11T04:28:08.148 INFO:tasks.ceph:[osd] err to stderr = True 2023-10-11T04:28:08.148 INFO:tasks.ceph:[osd] log flush on exit = True 2023-10-11T04:28:08.149 INFO:tasks.ceph:[osd] log to file = False 2023-10-11T04:28:08.149 INFO:tasks.ceph:[osd] log to stderr = True 2023-10-11T04:28:08.149 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-11T04:28:08.149 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-11T04:28:08.150 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-11T04:28:08.150 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-11T04:28:08.150 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-11T04:28:08.150 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-11T04:28:08.151 INFO:tasks.ceph:[global] auth client required = none 2023-10-11T04:28:08.151 INFO:tasks.ceph:[global] auth cluster required = none 2023-10-11T04:28:08.151 INFO:tasks.ceph:[global] auth service required = none 2023-10-11T04:28:08.151 INFO:tasks.ceph:[global] auth supported = none 2023-10-11T04:28:08.152 INFO:tasks.ceph:[global] debug asok = 0/0 2023-10-11T04:28:08.152 INFO:tasks.ceph:[global] debug auth = 0/0 2023-10-11T04:28:08.152 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-10-11T04:28:08.152 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-10-11T04:28:08.153 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-10-11T04:28:08.153 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-10-11T04:28:08.153 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-10-11T04:28:08.153 INFO:tasks.ceph:[global] debug client = 0/0 2023-10-11T04:28:08.165 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-10-11T04:28:08.165 INFO:tasks.ceph:[global] debug context = 0/0 2023-10-11T04:28:08.166 INFO:tasks.ceph:[global] debug crush = 0/0 2023-10-11T04:28:08.166 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-10-11T04:28:08.166 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-10-11T04:28:08.167 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-10-11T04:28:08.167 INFO:tasks.ceph:[global] debug filer = 0/0 2023-10-11T04:28:08.167 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-10-11T04:28:08.167 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-10-11T04:28:08.168 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-10-11T04:28:08.168 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-10-11T04:28:08.168 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-10-11T04:28:08.168 INFO:tasks.ceph:[global] debug journal = 0/0 2023-10-11T04:28:08.169 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-10-11T04:28:08.169 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-10-11T04:28:08.169 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-10-11T04:28:08.169 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-10-11T04:28:08.170 INFO:tasks.ceph:[global] debug mds = 0/0 2023-10-11T04:28:08.170 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-10-11T04:28:08.170 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-10-11T04:28:08.170 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-10-11T04:28:08.171 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-10-11T04:28:08.171 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-10-11T04:28:08.171 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-10-11T04:28:08.171 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-10-11T04:28:08.172 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-10-11T04:28:08.172 INFO:tasks.ceph:[global] debug mon = 0/0 2023-10-11T04:28:08.172 INFO:tasks.ceph:[global] debug monc = 0/0 2023-10-11T04:28:08.172 INFO:tasks.ceph:[global] debug ms = 0/0 2023-10-11T04:28:08.173 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-10-11T04:28:08.173 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-10-11T04:28:08.173 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-10-11T04:28:08.173 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-10-11T04:28:08.174 INFO:tasks.ceph:[global] debug osd = 0/0 2023-10-11T04:28:08.174 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-10-11T04:28:08.174 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-10-11T04:28:08.174 INFO:tasks.ceph:[global] debug rados = 0/0 2023-10-11T04:28:08.175 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-10-11T04:28:08.175 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-10-11T04:28:08.175 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-10-11T04:28:08.175 INFO:tasks.ceph:[global] debug refs = 0/0 2023-10-11T04:28:08.176 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-10-11T04:28:08.176 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-10-11T04:28:08.176 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-10-11T04:28:08.176 INFO:tasks.ceph:[global] debug striper = 0/0 2023-10-11T04:28:08.177 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-10-11T04:28:08.177 INFO:tasks.ceph:[global] debug timer = 0/0 2023-10-11T04:28:08.177 INFO:tasks.ceph:[global] debug tp = 0/0 2023-10-11T04:28:08.177 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-11T04:28:08.178 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-11T04:28:08.178 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-11T04:28:08.178 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-11T04:28:08.179 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-11T04:28:08.179 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-11T04:28:08.179 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-11T04:28:08.180 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-11T04:28:08.180 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-11T04:28:08.180 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-10-11T04:28:08.181 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-10-11T04:28:08.181 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-10-11T04:28:08.181 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-11T04:28:08.182 INFO:tasks.ceph:Setting up mon.a... 2023-10-11T04:28:08.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-11T04:28:08.242 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2023-10-11T04:28:08.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-11T04:28:08.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-11T04:28:08.330 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40')] 2023-10-11T04:28:08.330 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-11T04:28:08.332 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:08.332 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T04:28:08.384 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:08.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:08.469+0000 7f7fe402ee80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:08.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:08.469+0000 7f7fe402ee80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:08.469+0000 7f7fe402ee80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:08.472 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:08.472 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363 2023-10-11T04:28:08.472 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = pacific 2023-10-11T04:28:08.472 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2023-10-11T04:28:08.473 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363 2023-10-11T04:28:08.473 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2023-10-11T04:28:08.470620+0000 2023-10-11T04:28:08.473 INFO:teuthology.orchestra.run.smithi040.stdout:created 2023-10-11T04:28:08.470620+0000 2023-10-11T04:28:08.473 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 16 (pacific) 2023-10-11T04:28:08.473 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2023-10-11T04:28:08.474 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-10-11T04:28:08.474 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-11T04:28:08.474 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-11T04:28:08.489 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363... 2023-10-11T04:28:08.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-11T04:28:08.590 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2023-10-11T04:28:08.590 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2023-10-11T04:28:08.590 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-11T04:28:08.591 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = none 2023-10-11T04:28:08.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.591 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2023-10-11T04:28:08.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.591 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2023-10-11T04:28:08.592 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.592 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2023-10-11T04:28:08.592 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2023-10-11T04:28:08.592 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.592 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2023-10-11T04:28:08.593 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2023-10-11T04:28:08.593 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-11T04:28:08.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2023-10-11T04:28:08.593 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.594 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2023-10-11T04:28:08.594 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-11T04:28:08.594 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.594 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2023-10-11T04:28:08.594 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2023-10-11T04:28:08.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2023-10-11T04:28:08.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2023-10-11T04:28:08.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2023-10-11T04:28:08.595 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2023-10-11T04:28:08.596 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2023-10-11T04:28:08.596 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2023-10-11T04:28:08.596 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-11T04:28:08.596 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-11T04:28:08.596 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2023-10-11T04:28:08.596 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.597 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-11T04:28:08.597 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.597 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2023-10-11T04:28:08.597 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.597 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2023-10-11T04:28:08.598 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.598 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2023-10-11T04:28:08.598 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2023-10-11T04:28:08.598 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2023-10-11T04:28:08.598 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = 172.21.15.40 2023-10-11T04:28:08.599 INFO:teuthology.orchestra.run.smithi040.stdout: auth client required = none 2023-10-11T04:28:08.599 INFO:teuthology.orchestra.run.smithi040.stdout: auth cluster required = none 2023-10-11T04:28:08.599 INFO:teuthology.orchestra.run.smithi040.stdout: auth service required = none 2023-10-11T04:28:08.599 INFO:teuthology.orchestra.run.smithi040.stdout: debug asok = 0/0 2023-10-11T04:28:08.599 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 0/0 2023-10-11T04:28:08.600 INFO:teuthology.orchestra.run.smithi040.stdout: debug bdev = 0/0 2023-10-11T04:28:08.600 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 0/0 2023-10-11T04:28:08.600 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 0/0 2023-10-11T04:28:08.600 INFO:teuthology.orchestra.run.smithi040.stdout: debug buffer = 0/0 2023-10-11T04:28:08.600 INFO:teuthology.orchestra.run.smithi040.stdout: debug civetweb = 0/0 2023-10-11T04:28:08.600 INFO:teuthology.orchestra.run.smithi040.stdout: debug client = 0/0 2023-10-11T04:28:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: debug compressor = 0/0 2023-10-11T04:28:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: debug context = 0/0 2023-10-11T04:28:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: debug crush = 0/0 2023-10-11T04:28:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: debug crypto = 0/0 2023-10-11T04:28:08.601 INFO:teuthology.orchestra.run.smithi040.stdout: debug dpdk = 0/0 2023-10-11T04:28:08.602 INFO:teuthology.orchestra.run.smithi040.stdout: debug eventtrace = 0/0 2023-10-11T04:28:08.602 INFO:teuthology.orchestra.run.smithi040.stdout: debug filer = 0/0 2023-10-11T04:28:08.603 INFO:teuthology.orchestra.run.smithi040.stdout: debug filestore = 0/0 2023-10-11T04:28:08.603 INFO:teuthology.orchestra.run.smithi040.stdout: debug finisher = 0/0 2023-10-11T04:28:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: debug fuse = 0/0 2023-10-11T04:28:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: debug heartbeatmap = 0/0 2023-10-11T04:28:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: debug javaclient = 0/0 2023-10-11T04:28:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: debug journal = 0/0 2023-10-11T04:28:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: debug journaler = 0/0 2023-10-11T04:28:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: debug kstore = 0/0 2023-10-11T04:28:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: debug leveldb = 0/0 2023-10-11T04:28:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: debug lockdep = 0/0 2023-10-11T04:28:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds = 0/0 2023-10-11T04:28:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds balancer = 0/0 2023-10-11T04:28:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds locker = 0/0 2023-10-11T04:28:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds log = 0/0 2023-10-11T04:28:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds log expire = 0/0 2023-10-11T04:28:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds migrator = 0/0 2023-10-11T04:28:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: debug memdb = 0/0 2023-10-11T04:28:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 0/0 2023-10-11T04:28:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgrc = 0/0 2023-10-11T04:28:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 0/0 2023-10-11T04:28:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 0/0 2023-10-11T04:28:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 0/0 2023-10-11T04:28:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: debug objclass = 0/0 2023-10-11T04:28:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: debug objectcacher = 0/0 2023-10-11T04:28:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: debug objecter = 0/0 2023-10-11T04:28:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: debug optracker = 0/0 2023-10-11T04:28:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 0/0 2023-10-11T04:28:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 0/0 2023-10-11T04:28:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug perfcounter = 0/0 2023-10-11T04:28:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug rados = 0/0 2023-10-11T04:28:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug rbd = 0/0 2023-10-11T04:28:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug rbd mirror = 0/0 2023-10-11T04:28:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug rbd replay = 0/0 2023-10-11T04:28:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug refs = 0/0 2023-10-11T04:28:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug rgw = 0/0 2023-10-11T04:28:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug rgw sync = 0/0 2023-10-11T04:28:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 0/0 2023-10-11T04:28:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug striper = 0/0 2023-10-11T04:28:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: debug throttle = 0/0 2023-10-11T04:28:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: debug timer = 0/0 2023-10-11T04:28:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: debug tp = 0/0 2023-10-11T04:28:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-11T04:28:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: ms client mode = crc 2023-10-11T04:28:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: ms cluster mode = crc 2023-10-11T04:28:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon client mode = crc 2023-10-11T04:28:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon cluster mode = crc 2023-10-11T04:28:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon service mode = crc 2023-10-11T04:28:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: ms service mode = crc 2023-10-11T04:28:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363 2023-10-11T04:28:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2023-10-11T04:28:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2023-10-11T04:28:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2023-10-11T04:28:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2023-10-11T04:28:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2023-10-11T04:28:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2023-10-11T04:28:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2023-10-11T04:28:08.617 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.617 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2023-10-11T04:28:08.617 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2023-10-11T04:28:08.618 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2023-10-11T04:28:08.618 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.618 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2023-10-11T04:28:08.618 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2023-10-11T04:28:08.619 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.619 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2023-10-11T04:28:08.619 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.619 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2023-10-11T04:28:08.620 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.620 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2023-10-11T04:28:08.620 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.620 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2023-10-11T04:28:08.620 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2023-10-11T04:28:08.621 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.621 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2023-10-11T04:28:08.621 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2023-10-11T04:28:08.622 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2023-10-11T04:28:08.622 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2023-10-11T04:28:08.623 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression algorithm = snappy 2023-10-11T04:28:08.624 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression mode = aggressive 2023-10-11T04:28:08.624 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2023-10-11T04:28:08.624 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore rocksdb cf = False 2023-10-11T04:28:08.624 INFO:teuthology.orchestra.run.smithi040.stdout: crimson seastar smp = 3 2023-10-11T04:28:08.624 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 20 2023-10-11T04:28:08.625 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 20 2023-10-11T04:28:08.625 INFO:teuthology.orchestra.run.smithi040.stdout: debug filestore = 0/0 2023-10-11T04:28:08.625 INFO:teuthology.orchestra.run.smithi040.stdout: debug journal = 0/0 2023-10-11T04:28:08.625 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 0/0 2023-10-11T04:28:08.625 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 0/0 2023-10-11T04:28:08.626 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2023-10-11T04:28:08.626 INFO:teuthology.orchestra.run.smithi040.stdout: err to stderr = True 2023-10-11T04:28:08.626 INFO:teuthology.orchestra.run.smithi040.stdout: log flush on exit = True 2023-10-11T04:28:08.626 INFO:teuthology.orchestra.run.smithi040.stdout: log to file = False 2023-10-11T04:28:08.626 INFO:teuthology.orchestra.run.smithi040.stdout: log to stderr = True 2023-10-11T04:28:08.627 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2023-10-11T04:28:08.627 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2023-10-11T04:28:08.627 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2023-10-11T04:28:08.627 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2023-10-11T04:28:08.627 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2023-10-11T04:28:08.628 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2023-10-11T04:28:08.628 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.628 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2023-10-11T04:28:08.628 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-10-11T04:28:08.628 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2023-10-11T04:28:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-10-11T04:28:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-10-11T04:28:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-10-11T04:28:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-10-11T04:28:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2023-10-11T04:28:08.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.630 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2023-10-11T04:28:08.630 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 0/0 2023-10-11T04:28:08.630 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 0/0 2023-10-11T04:28:08.631 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 0/0 2023-10-11T04:28:08.631 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-10-11T04:28:08.631 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2023-10-11T04:28:08.631 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2023-10-11T04:28:08.631 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-10-11T04:28:08.632 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2023-10-11T04:28:08.632 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2023-10-11T04:28:08.632 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-10-11T04:28:08.632 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.632 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-11T04:28:08.633 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2023-10-11T04:28:08.633 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2023-10-11T04:28:08.633 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.633 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2023-10-11T04:28:08.633 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-11T04:28:08.633 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-11T04:28:08.634 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.634 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2023-10-11T04:28:08.634 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2023-10-11T04:28:08.634 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.634 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2023-10-11T04:28:08.635 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default crimson = True 2023-10-11T04:28:08.635 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:08.635 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2023-10-11T04:28:08.635 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2023-10-11T04:28:08.635 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2023-10-11T04:28:08.636 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2023-10-11T04:28:08.636 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-11T04:28:08.636 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-11T04:28:08.636 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2023-10-11T04:28:08.637 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-11T04:28:08.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-11T04:28:08.703 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-11T04:28:08.703 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:08.703 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-11T04:28:08.719 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:08.719 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-11T04:28:08.774 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:28:08.774 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:08.775 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-11T04:28:08.775 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-11T04:28:08.861 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:08.861 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:08.916 INFO:tasks.ceph:Setting up mon nodes... 2023-10-11T04:28:08.917 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-11T04:28:08.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-11T04:28:09.023 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-11T04:28:09.026 INFO:tasks.ceph:Setting up mds nodes... 2023-10-11T04:28:09.026 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-11T04:28:09.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-11T04:28:09.078 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-11T04:28:09.098 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-11T04:28:09.099 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-11T04:28:09.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-11T04:28:09.131 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T04:28:09.132 INFO:tasks.ceph:role: osd.0 2023-10-11T04:28:09.132 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:28:09.132 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-11T04:28:09.209 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T04:28:09.209 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T04:28:09.209 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T04:28:09.210 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-11T04:28:09.210 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T04:28:09.210 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-11T04:28:09.210 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T04:28:09.210 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-11T04:28:09.211 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T04:28:09.211 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T04:28:09.726 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-11T04:28:09.728 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-11T04:28:09.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-11T04:28:09.884 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-11T04:28:10.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-11T04:28:10.221 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T04:28:10.222 INFO:tasks.ceph:role: osd.1 2023-10-11T04:28:10.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:28:10.222 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-11T04:28:10.301 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T04:28:10.301 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T04:28:10.301 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T04:28:10.302 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-11T04:28:10.302 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T04:28:10.302 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-11T04:28:10.302 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T04:28:10.303 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-11T04:28:10.303 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T04:28:10.303 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T04:28:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-11T04:28:10.791 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-11T04:28:10.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-11T04:28:10.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-11T04:28:10.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-11T04:28:11.007 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-11T04:28:11.008 INFO:tasks.ceph:role: osd.2 2023-10-11T04:28:11.008 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:28:11.008 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-11T04:28:11.087 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-11T04:28:11.088 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-11T04:28:11.088 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-11T04:28:11.088 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-11T04:28:11.088 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-11T04:28:11.088 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-10-11T04:28:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-11T04:28:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-11T04:28:11.089 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-11T04:28:11.089 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-11T04:28:11.607 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-10-11T04:28:11.609 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-10-11T04:28:11.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-11T04:28:11.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-11T04:28:11.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:12.345 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-11T04:28:12.345 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,345 seastar - Reactor backend: linux-aio 2023-10-11T04:28:12.361 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,361 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-11T04:28:12.361 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,361 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-11T04:28:12.362 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,361 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-11T04:28:12.362 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,361 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-11T04:28:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:==79792==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-11T04:28:12.370 INFO:teuthology.orchestra.run.smithi040.stderr:==79792==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff95349000; bottom 0x7f7521c32000; size: 0x008a73717000 (594642300928) 2023-10-11T04:28:12.370 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2023-10-11T04:28:12.370 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-11T04:28:12.925 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:12.925 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2023-10-11T04:28:12.925 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2023-10-11T04:28:12.925 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2023-10-11T04:28:12.926 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:12.926 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:12.979 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-11T04:28:12.980 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,980 seastar - Reactor backend: linux-aio 2023-10-11T04:28:12.997 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,997 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-11T04:28:12.998 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,997 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-11T04:28:12.998 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:12,997 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-11T04:28:13.005 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:13,005 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-11T04:28:13.007 INFO:teuthology.orchestra.run.smithi040.stderr:==79790==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-11T04:28:13.014 INFO:teuthology.orchestra.run.smithi040.stderr:==79790==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff95349000; bottom 0x7f751fa1b000; size: 0x008a7592e000 (594678046720) 2023-10-11T04:28:13.014 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2023-10-11T04:28:13.014 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-11T04:28:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c844dd6 0x2c846184 0x2c8476d2 0x2c847a56 0x2c847ba0 0x2c847ff6 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2640d0dc 0x2640f725 0x26411536 0x26413d9b 0x2624394f 0x1ee15204 0x1ed896f5 0x1ed8d661 0x2cbc8a39 2023-10-11T04:28:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2023-10-11T04:28:17.045 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:17.045 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2023-10-11T04:28:17.046 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2023-10-11T04:28:17.046 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2023-10-11T04:28:17.046 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:17.046 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:17.047 INFO:teuthology.orchestra.run.smithi040.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363 2023-10-11T04:28:17.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-11T04:28:17.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:17.656 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-11T04:28:17.657 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:17,657 seastar - Reactor backend: linux-aio 2023-10-11T04:28:17.673 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:17,673 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-11T04:28:17.673 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:17,673 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-11T04:28:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:17,673 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-11T04:28:17.674 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:17,673 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-11T04:28:17.675 INFO:teuthology.orchestra.run.smithi040.stderr:==79974==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-11T04:28:17.681 INFO:teuthology.orchestra.run.smithi040.stderr:==79974==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffce7b8000; bottom 0x7f672e2a9000; size: 0x0098a050f000 (655524687872) 2023-10-11T04:28:17.682 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2023-10-11T04:28:17.682 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-11T04:28:18.229 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:18.230 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2023-10-11T04:28:18.230 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2023-10-11T04:28:18.230 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2023-10-11T04:28:18.230 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:18.231 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:18.284 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-11T04:28:18.285 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:18,285 seastar - Reactor backend: linux-aio 2023-10-11T04:28:18.303 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:18,302 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-11T04:28:18.303 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:18,302 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-11T04:28:18.304 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:18,303 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-11T04:28:18.310 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:18,310 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-11T04:28:18.312 INFO:teuthology.orchestra.run.smithi040.stderr:==79972==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-11T04:28:18.319 INFO:teuthology.orchestra.run.smithi040.stderr:==79972==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffce7b8000; bottom 0x7f672c092000; size: 0x0098a2726000 (655560433664) 2023-10-11T04:28:18.319 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2023-10-11T04:28:18.319 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-11T04:28:18.345 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c844dd6 0x2c846184 0x2c8476d2 0x2c847a56 0x2c847ba0 0x2c847ff6 0x54daf 0x22b9b2b6 0x22ba842f 0x22bae47b 0x22bafe3a 0x24ba566e 0x24baa706 0x24bad054 0x24bae22d 0x22774737 0x227c88fe 0x227e84fb 0x2c7e70f1 0x2c884f0c 0x2caa8d0e 0x2caaab59 0x2c5030d4 0x2c504fd5 0x1ed5afe7 0x3feaf 0x3ff5f 0x1e812a04 2023-10-11T04:28:18.345 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2023-10-11T04:28:22.332 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:22.332 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2023-10-11T04:28:22.333 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2023-10-11T04:28:22.333 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2023-10-11T04:28:22.333 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:22.333 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:22.333 INFO:teuthology.orchestra.run.smithi040.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363 2023-10-11T04:28:22.384 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-11T04:28:22.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:22.965 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-11T04:28:22.966 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:22,966 seastar - Reactor backend: linux-aio 2023-10-11T04:28:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:22,986 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-11T04:28:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:22,986 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-11T04:28:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:22,986 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-11T04:28:22.987 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:22,986 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-11T04:28:22.988 INFO:teuthology.orchestra.run.smithi040.stderr:==80156==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-11T04:28:22.995 INFO:teuthology.orchestra.run.smithi040.stderr:==80156==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff9866c000; bottom 0x7fcaa0ecf000; size: 0x0034f779d000 (227490254848) 2023-10-11T04:28:22.995 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2023-10-11T04:28:22.995 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-11T04:28:23.021 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c844dd6 0x2c846184 0x2c8476d2 0x2c847a56 0x2c847ba0 0x2c847ff6 0x54daf 0x22eafcf8 0x22ed39fa 0x22eb23fc 0x22ec097c 0x22ec205b 0x22f1b906 0x22f1c9cc 0x22ed6892 0x22f1f70e 0x22f2163d 0x22f23042 0x22f23da0 0x22e9dbbc 0x22d649ec 0x22be6366 0x24ba4ba1 0x24baa706 0x24bad054 0x24bae22d 0x22774737 0x227c88fe 0x227e84fb 0x2c7e70f1 0x2c884f0c 0x2caa8d0e 0x2caaab59 0x2c5030d4 0x2c504fd5 0x1eed4f0c 0x1eedf079 0x1ed59731 0x3feaf 0x3ff5f 0x1e812a04 2023-10-11T04:28:23.021 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2023-10-11T04:28:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:23.532 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2023-10-11T04:28:23.532 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2023-10-11T04:28:23.533 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2023-10-11T04:28:23.533 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:23.533 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:23.592 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-11T04:28:23.592 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:23,593 seastar - Reactor backend: linux-aio 2023-10-11T04:28:23.611 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:23,610 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-11T04:28:23.611 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:23,610 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-11T04:28:23.611 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:23,610 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-11T04:28:23.618 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2023-10-11 04:28:23,618 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-11T04:28:23.621 INFO:teuthology.orchestra.run.smithi040.stderr:==80154==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-11T04:28:23.627 INFO:teuthology.orchestra.run.smithi040.stderr:==80154==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff9866c000; bottom 0x7fca9ecb8000; size: 0x0034f99b4000 (227526000640) 2023-10-11T04:28:23.627 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2023-10-11T04:28:23.627 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-11T04:28:23.654 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c844dd6 0x2c846184 0x2c8476d2 0x2c847a56 0x2c847ba0 0x2c847ff6 0x54daf 0x22b42a1a 0x22bb0135 0x24ba566e 0x24baa706 0x24bad054 0x24bae22d 0x22774737 0x227c88fe 0x227e84fb 0x2c7e70f1 0x2c884f0c 0x2caa8d0e 0x2caaab59 0x2c5030d4 0x2c504fd5 0x1ed5afe7 0x3feaf 0x3ff5f 0x1e812a04 2023-10-11T04:28:23.655 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2023-10-11T04:28:27.673 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:27.673 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2023-10-11T04:28:27.673 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2023-10-11T04:28:27.674 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2023-10-11T04:28:27.674 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2023-10-11T04:28:27.674 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:27.675 INFO:teuthology.orchestra.run.smithi040.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 0807a2b3-2dc9-46c9-b5fe-7ea6ee978363 2023-10-11T04:28:27.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-11T04:28:27.741 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-11T04:28:27.741 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:27.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-11T04:28:27.812 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:27.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-11T04:28:27.881 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:27.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-11T04:28:27.952 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:27.952 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-11T04:28:28.023 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:28.023 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-11T04:28:28.079 INFO:tasks.ceph:Adding keys to all mons... 2023-10-11T04:28:28.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-11T04:28:28.147 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2023-10-11T04:28:28.148 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBZJCZlyaJqARAA10jzz7Iaj6ZZaDxdrBF/eg== 2023-10-11T04:28:28.148 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2023-10-11T04:28:28.148 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBdJCZlt5DBAxAAWU/5BXAN5vniWl3vOXvwLA== 2023-10-11T04:28:28.148 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2023-10-11T04:28:28.148 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBiJCZlARDhFRAA0zyi7OwhcLbwEUspKePqag== 2023-10-11T04:28:28.149 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2023-10-11T04:28:28.149 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBnJCZlTohKKBAAiQSfSR3zSmpoALGW2ijoqA== 2023-10-11T04:28:28.149 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2023-10-11T04:28:28.149 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQBZJCZlCrOzBBAASiabihBZTfovXNwu1gt5Qw== 2023-10-11T04:28:28.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-11T04:28:28.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T04:28:28.314 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T04:28:28.376 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-11T04:28:28.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-11T04:28:28.504 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-11T04:28:28.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-11T04:28:28.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-11T04:28:28.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:28.638+0000 7f79f482a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:28.638+0000 7f79f482a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.657 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-11T04:28:28.687 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-11T04:28:28.742 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-11T04:28:28.743 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-11T04:28:28.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-11T04:28:28.785 INFO:tasks.ceph.mon.a:Started 2023-10-11T04:28:28.785 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-11T04:28:28.786 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-11T04:28:28.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-11T04:28:28.789 INFO:tasks.ceph.mgr.x:Started 2023-10-11T04:28:28.790 INFO:tasks.ceph:Setting crush tunables to default 2023-10-11T04:28:28.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2023-10-11T04:28:28.869 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-10-11T04:28:28.867+0000 7f72e9127980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-10-11T04:28:28.867+0000 7f72e9127980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.870 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-10-11T04:28:28.868+0000 7f72e9127980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.873 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:28.872+0000 7f3f1983af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:28.900+0000 7f6a4442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:28.927+0000 7f6a4442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:28.998 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2023-10-11T04:28:29.009 INFO:tasks.ceph:check_enable_crimson: True 2023-10-11T04:28:29.010 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-11T04:28:29.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-11T04:28:29.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:29.075+0000 7f3f1983af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:29.077 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:29.075+0000 7f3f1983af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:29.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:29.114+0000 7fb34091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:29.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:29.120+0000 7fb34091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:29.202 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-10-11T04:28:29.200+0000 7f72e13f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-11T04:28:29.202 INFO:tasks.ceph.mon.a.smithi040.stderr:Please be aware that this feature is experimental, untested, 2023-10-11T04:28:29.203 INFO:tasks.ceph.mon.a.smithi040.stderr:unsupported, and may result in data corruption, data loss, 2023-10-11T04:28:29.203 INFO:tasks.ceph.mon.a.smithi040.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-11T04:28:29.203 INFO:tasks.ceph.mon.a.smithi040.stderr:feature with important data. 2023-10-11T04:28:29.203 INFO:tasks.ceph.mon.a.smithi040.stderr: 2023-10-11T04:28:29.584 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:29.582+0000 7f3f1983af00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-11T04:28:29.637 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:29.635+0000 7f3f1983af00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-11T04:28:30.014 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-11T04:28:30.014 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:30.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-11T04:28:30.046 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:30.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-11T04:28:30.117 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:30.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-11T04:28:30.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 44ef8274-1eb7-48ed-8bc6-2ac77e461924 0 2023-10-11T04:28:30.191 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.189+0000 7f3f1983af00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-11T04:28:30.247 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.245+0000 7f3f1983af00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-11T04:28:30.296 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.294+0000 7f3f1983af00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-11T04:28:30.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:30.295+0000 7f99d699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:30.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:30.302+0000 7f99d699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:30.353 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.350+0000 7f3f1983af00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-11T04:28:30.382 INFO:teuthology.orchestra.run.smithi040.stdout:0 2023-10-11T04:28:30.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 4e8707c3-25e3-452e-85ef-8880c62a6931 1 2023-10-11T04:28:30.397 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.396+0000 7f3f1983af00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-11T04:28:30.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:30.497+0000 7f603b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:30.505 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.504+0000 7f3f1983af00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-11T04:28:30.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:30.505+0000 7f603b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:30.579 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.577+0000 7f3f1983af00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-11T04:28:30.585 INFO:teuthology.orchestra.run.smithi040.stdout:1 2023-10-11T04:28:30.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd new 109f151d-b2ee-4218-85cf-83d7d7a86af7 2 2023-10-11T04:28:30.628 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.626+0000 7f3f1983af00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-11T04:28:30.683 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.681+0000 7f3f1983af00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-11T04:28:30.734 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:30.733+0000 7f3f1983af00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-11T04:28:30.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:30.732+0000 7f6a3a35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:30.739+0000 7f6a3a35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:30.820 INFO:teuthology.orchestra.run.smithi040.stdout:2 2023-10-11T04:28:30.831 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-11T04:28:30.866 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-11T04:28:30.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-11T04:28:30.909 INFO:tasks.ceph.osd.0:Started 2023-10-11T04:28:30.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-11T04:28:30.945 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-11T04:28:30.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-11T04:28:30.988 INFO:tasks.ceph.osd.1:Started 2023-10-11T04:28:30.988 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-11T04:28:31.028 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-11T04:28:31.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-11T04:28:31.071 INFO:tasks.ceph.osd.2:Started 2023-10-11T04:28:31.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T04:28:31.179 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.177+0000 7f3f1983af00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-11T04:28:31.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:31.291+0000 7ff82879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:31.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:31.302+0000 7ff82879e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:31.352 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.350+0000 7f3f1983af00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-11T04:28:31.443 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:31.443 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":6,"fsid":"0807a2b3-2dc9-46c9-b5fe-7ea6ee978363","created":"2023-10-11T04:28:28.920128+0000","modified":"2023-10-11T04:28:30.819360+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T04:28:30.819360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"44ef8274-1eb7-48ed-8bc6-2ac77e461924","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e8707c3-25e3-452e-85ef-8880c62a6931","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"109f151d-b2ee-4218-85cf-83d7d7a86af7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T04:28:31.466 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-11T04:28:31.467 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-11T04:28:31.487 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.485+0000 7f3f1983af00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-11T04:28:31.539 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.538+0000 7f3f1983af00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-11T04:28:31.645 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.643+0000 7f3f1983af00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-11T04:28:31.746 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.744+0000 7f3f1983af00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-11T04:28:31.770 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-11T04:28:31.870 INFO:teuthology.misc.health.smithi040.stderr:2023-10-11T04:28:31.864+0000 7fb03c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:31.872 INFO:teuthology.misc.health.smithi040.stderr:2023-10-11T04:28:31.870+0000 7fb03c86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:31.919 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.917+0000 7f3f1983af00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-11T04:28:31.959 INFO:teuthology.misc.health.smithi040.stdout: 2023-10-11T04:28:31.960 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":6,"fsid":"0807a2b3-2dc9-46c9-b5fe-7ea6ee978363","created":"2023-10-11T04:28:28.920128+0000","modified":"2023-10-11T04:28:30.819360+0000","last_up_change":"0.000000","last_in_change":"2023-10-11T04:28:30.819360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"44ef8274-1eb7-48ed-8bc6-2ac77e461924","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e8707c3-25e3-452e-85ef-8880c62a6931","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"109f151d-b2ee-4218-85cf-83d7d7a86af7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T04:28:31.967 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:31.965+0000 7f3f1983af00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-11T04:28:31.971 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-11T04:28:32.123 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:32.121+0000 7f3f1983af00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-11T04:28:32.988 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:32.986+0000 7f3f1983af00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-11T04:28:33.276 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:33.274+0000 7f3f1983af00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-11T04:28:33.330 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:33.329+0000 7f3f1983af00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-11T04:28:33.437 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:33.435+0000 7f3f1983af00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-11T04:28:34.456 INFO:tasks.ceph.mgr.x.smithi040.stderr:2023-10-11T04:28:34.455+0000 7f3eb9927640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-11T04:28:38.274 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-11T04:28:38.384 INFO:teuthology.misc.health.smithi040.stderr:2023-10-11T04:28:38.380+0000 7f5084a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:38.386 INFO:teuthology.misc.health.smithi040.stderr:2023-10-11T04:28:38.385+0000 7f5084a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:38.573 INFO:teuthology.misc.health.smithi040.stdout: 2023-10-11T04:28:38.573 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":9,"fsid":"0807a2b3-2dc9-46c9-b5fe-7ea6ee978363","created":"2023-10-11T04:28:28.920128+0000","modified":"2023-10-11T04:28:38.463092+0000","last_up_change":"2023-10-11T04:28:36.456886+0000","last_in_change":"2023-10-11T04:28:30.819360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T04:28:37.507888+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"44ef8274-1eb7-48ed-8bc6-2ac77e461924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":557095925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":557095925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":557095925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":557095925}]},"public_addr":"172.21.15.40:6804/557095925","cluster_addr":"172.21.15.40:6805/557095925","heartbeat_back_addr":"172.21.15.40:6809/557095925","heartbeat_front_addr":"172.21.15.40:6808/557095925","state":["exists","up"]},{"osd":1,"uuid":"4e8707c3-25e3-452e-85ef-8880c62a6931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":3408338388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3408338388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":3408338388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3408338388}]},"public_addr":"172.21.15.40:6803/3408338388","cluster_addr":"172.21.15.40:6802/3408338388","heartbeat_back_addr":"172.21.15.40:6811/3408338388","heartbeat_front_addr":"172.21.15.40:6810/3408338388","state":["exists","up"]},{"osd":2,"uuid":"109f151d-b2ee-4218-85cf-83d7d7a86af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2766121097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":2766121097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":2766121097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2766121097}]},"public_addr":"172.21.15.40:6806/2766121097","cluster_addr":"172.21.15.40:6807/2766121097","heartbeat_back_addr":"172.21.15.40:6813/2766121097","heartbeat_front_addr":"172.21.15.40:6812/2766121097","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T04:28:38.583 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-11T04:28:38.584 INFO:tasks.ceph:Creating RBD pool 2023-10-11T04:28:38.584 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-11T04:28:38.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:38.688+0000 7fb413d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:38.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:38.696+0000 7fb413d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:39.469 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'rbd' created 2023-10-11T04:28:39.487 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-11T04:28:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:39.604+0000 7fb40a71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:39.615+0000 7fb40a71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:40.476 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-11T04:28:40.489 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-11T04:28:40.490 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-11T04:28:40.490 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-11T04:28:40.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-11T04:28:40.491 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-11T04:28:40.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:40.610+0000 7ff21451e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:40.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:40.616+0000 7ff21451e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:40.857 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:40.868 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2103148538},{"type":"v1","addr":"172.21.15.40:6801","nonce":2103148538}]},"active_addr":"172.21.15.40:6801/2103148538","active_change":"2023-10-11T04:28:33.449380+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":3311772402}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2635811557}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2529777356}]}]} 2023-10-11T04:28:40.870 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-11T04:28:40.871 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-11T04:28:40.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T04:28:40.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:40.977+0000 7fcc6926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:40.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:40.982+0000 7fcc6926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.170 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:41.171 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":11,"fsid":"0807a2b3-2dc9-46c9-b5fe-7ea6ee978363","created":"2023-10-11T04:28:28.920128+0000","modified":"2023-10-11T04:28:40.473359+0000","last_up_change":"2023-10-11T04:28:36.456886+0000","last_in_change":"2023-10-11T04:28:30.819360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T04:28:37.507888+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T04:28:38.908646+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"44ef8274-1eb7-48ed-8bc6-2ac77e461924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":557095925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":557095925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":557095925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":557095925}]},"public_addr":"172.21.15.40:6804/557095925","cluster_addr":"172.21.15.40:6805/557095925","heartbeat_back_addr":"172.21.15.40:6809/557095925","heartbeat_front_addr":"172.21.15.40:6808/557095925","state":["exists","up"]},{"osd":1,"uuid":"4e8707c3-25e3-452e-85ef-8880c62a6931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":3408338388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3408338388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":3408338388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3408338388}]},"public_addr":"172.21.15.40:6803/3408338388","cluster_addr":"172.21.15.40:6802/3408338388","heartbeat_back_addr":"172.21.15.40:6811/3408338388","heartbeat_front_addr":"172.21.15.40:6810/3408338388","state":["exists","up"]},{"osd":2,"uuid":"109f151d-b2ee-4218-85cf-83d7d7a86af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2766121097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":2766121097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":2766121097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2766121097}]},"public_addr":"172.21.15.40:6806/2766121097","cluster_addr":"172.21.15.40:6807/2766121097","heartbeat_back_addr":"172.21.15.40:6813/2766121097","heartbeat_front_addr":"172.21.15.40:6812/2766121097","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T04:28:41.183 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-11T04:28:41.183 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-11T04:28:41.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.291+0000 7f9e5e224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.296+0000 7f9e5e224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:41.498 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"fsid":"0807a2b3-2dc9-46c9-b5fe-7ea6ee978363","created":"2023-10-11T04:28:28.920128+0000","modified":"2023-10-11T04:28:41.476575+0000","last_up_change":"2023-10-11T04:28:36.456886+0000","last_in_change":"2023-10-11T04:28:30.819360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-11T04:28:37.507888+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-11T04:28:38.908646+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"44ef8274-1eb7-48ed-8bc6-2ac77e461924","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":557095925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":557095925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":557095925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":557095925}]},"public_addr":"172.21.15.40:6804/557095925","cluster_addr":"172.21.15.40:6805/557095925","heartbeat_back_addr":"172.21.15.40:6809/557095925","heartbeat_front_addr":"172.21.15.40:6808/557095925","state":["exists","up"]},{"osd":1,"uuid":"4e8707c3-25e3-452e-85ef-8880c62a6931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":3408338388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3408338388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":3408338388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3408338388}]},"public_addr":"172.21.15.40:6803/3408338388","cluster_addr":"172.21.15.40:6802/3408338388","heartbeat_back_addr":"172.21.15.40:6811/3408338388","heartbeat_front_addr":"172.21.15.40:6810/3408338388","state":["exists","up"]},{"osd":2,"uuid":"109f151d-b2ee-4218-85cf-83d7d7a86af7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2766121097}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":2766121097}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":2766121097}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":2766121097}]},"public_addr":"172.21.15.40:6806/2766121097","cluster_addr":"172.21.15.40:6807/2766121097","heartbeat_back_addr":"172.21.15.40:6813/2766121097","heartbeat_front_addr":"172.21.15.40:6812/2766121097","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-11T04:28:41.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-11T04:28:41.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-11T04:28:41.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-11T04:28:41.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.630+0000 7f73c299e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.630+0000 7fed9899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.630+0000 7ff46e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.637+0000 7f73c299e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.637+0000 7fed9899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.637+0000 7ff46e82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.665 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-11T04:28:41.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-11T04:28:41.667 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-11T04:28:41.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-11T04:28:41.669 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-11T04:28:41.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-11T04:28:41.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.776+0000 7f8f5eb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.783+0000 7f8f5eb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.820+0000 7f81abade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.826+0000 7f81abade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.836+0000 7fd52579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:41.842+0000 7fd52579e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:42.004 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-11T04:28:42.012 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-11T04:28:42.017 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-11T04:28:42.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-11T04:28:42.038 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-11T04:28:42.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-11T04:28:43.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-11T04:28:43.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-11T04:28:43.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-11T04:28:43.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:43.147+0000 7f0cdbd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:43.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:43.155+0000 7f0cdbd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:43.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:43.182+0000 7fa583aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:43.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:43.182+0000 7f21c222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:43.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:43.188+0000 7fa583aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:43.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:43.189+0000 7f21c222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:43.345 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-11T04:28:43.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-11T04:28:43.376 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-11T04:28:43.379 INFO:teuthology.orchestra.run.smithi040.stdout:34359738372 2023-10-11T04:28:43.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-11T04:28:43.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-11T04:28:44.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-11T04:28:44.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-11T04:28:44.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-11T04:28:44.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.483+0000 7f66b226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.492+0000 7f66b226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.523+0000 7febf6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.523+0000 7f9c0fb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.529+0000 7febf6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.529+0000 7f9c0fb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.686 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-11T04:28:44.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-11T04:28:44.698 DEBUG:teuthology.parallel:result is None 2023-10-11T04:28:44.744 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-11T04:28:44.748 INFO:teuthology.orchestra.run.smithi040.stdout:34359738373 2023-10-11T04:28:44.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-11T04:28:44.757 DEBUG:teuthology.parallel:result is None 2023-10-11T04:28:44.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-11T04:28:44.761 DEBUG:teuthology.parallel:result is None 2023-10-11T04:28:44.761 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-11T04:28:44.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T04:28:44.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.910+0000 7f455d42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:44.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:44.917+0000 7f455d42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:45.101 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:45.102 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-11T04:28:45.113 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-11T04:28:43.453972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59922,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.942511"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.522103+0000","last_change":"2023-10-11T04:28:40.594324+0000","last_active":"2023-10-11T04:28:41.522103+0000","last_peered":"2023-10-11T04:28:41.522103+0000","last_clean":"2023-10-11T04:28:41.522103+0000","last_became_active":"2023-10-11T04:28:40.592168+0000","last_became_peered":"2023-10-11T04:28:40.592168+0000","last_unstale":"2023-10-11T04:28:41.522103+0000","last_undegraded":"2023-10-11T04:28:41.522103+0000","last_fullsized":"2023-10-11T04:28:41.522103+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.523765+0000","last_change":"2023-10-11T04:28:40.592383+0000","last_active":"2023-10-11T04:28:41.523765+0000","last_peered":"2023-10-11T04:28:41.523765+0000","last_clean":"2023-10-11T04:28:41.523765+0000","last_became_active":"2023-10-11T04:28:40.590469+0000","last_became_peered":"2023-10-11T04:28:40.590469+0000","last_unstale":"2023-10-11T04:28:41.523765+0000","last_undegraded":"2023-10-11T04:28:41.523765+0000","last_fullsized":"2023-10-11T04:28:41.523765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.523199+0000","last_change":"2023-10-11T04:28:40.573925+0000","last_active":"2023-10-11T04:28:41.523199+0000","last_peered":"2023-10-11T04:28:41.523199+0000","last_clean":"2023-10-11T04:28:41.523199+0000","last_became_active":"2023-10-11T04:28:40.561298+0000","last_became_peered":"2023-10-11T04:28:40.561298+0000","last_unstale":"2023-10-11T04:28:41.523199+0000","last_undegraded":"2023-10-11T04:28:41.523199+0000","last_fullsized":"2023-10-11T04:28:41.523199+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.511155+0000","last_change":"2023-10-11T04:28:40.606076+0000","last_active":"2023-10-11T04:28:41.511155+0000","last_peered":"2023-10-11T04:28:41.511155+0000","last_clean":"2023-10-11T04:28:41.511155+0000","last_became_active":"2023-10-11T04:28:40.602843+0000","last_became_peered":"2023-10-11T04:28:40.602843+0000","last_unstale":"2023-10-11T04:28:41.511155+0000","last_undegraded":"2023-10-11T04:28:41.511155+0000","last_fullsized":"2023-10-11T04:28:41.511155+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.527648+0000","last_change":"2023-10-11T04:28:40.584585+0000","last_active":"2023-10-11T04:28:41.527648+0000","last_peered":"2023-10-11T04:28:41.527648+0000","last_clean":"2023-10-11T04:28:41.527648+0000","last_became_active":"2023-10-11T04:28:40.582698+0000","last_became_peered":"2023-10-11T04:28:40.582698+0000","last_unstale":"2023-10-11T04:28:41.527648+0000","last_undegraded":"2023-10-11T04:28:41.527648+0000","last_fullsized":"2023-10-11T04:28:41.527648+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.486328+0000","last_change":"2023-10-11T04:28:40.557577+0000","last_active":"2023-10-11T04:28:41.486328+0000","last_peered":"2023-10-11T04:28:41.486328+0000","last_clean":"2023-10-11T04:28:41.486328+0000","last_became_active":"2023-10-11T04:28:40.555762+0000","last_became_peered":"2023-10-11T04:28:40.555762+0000","last_unstale":"2023-10-11T04:28:41.486328+0000","last_undegraded":"2023-10-11T04:28:41.486328+0000","last_fullsized":"2023-10-11T04:28:41.486328+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.486113+0000","last_change":"2023-10-11T04:28:40.551449+0000","last_active":"2023-10-11T04:28:41.486113+0000","last_peered":"2023-10-11T04:28:41.486113+0000","last_clean":"2023-10-11T04:28:41.486113+0000","last_became_active":"2023-10-11T04:28:40.549180+0000","last_became_peered":"2023-10-11T04:28:40.549180+0000","last_unstale":"2023-10-11T04:28:41.486113+0000","last_undegraded":"2023-10-11T04:28:41.486113+0000","last_fullsized":"2023-10-11T04:28:41.486113+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.516854+0000","last_change":"2023-10-11T04:28:40.559705+0000","last_active":"2023-10-11T04:28:41.516854+0000","last_peered":"2023-10-11T04:28:41.516854+0000","last_clean":"2023-10-11T04:28:41.516854+0000","last_became_active":"2023-10-11T04:28:40.556648+0000","last_became_peered":"2023-10-11T04:28:40.556648+0000","last_unstale":"2023-10-11T04:28:41.516854+0000","last_undegraded":"2023-10-11T04:28:41.516854+0000","last_fullsized":"2023-10-11T04:28:41.516854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.522101+0000","last_change":"2023-10-11T04:28:39.546833+0000","last_active":"2023-10-11T04:28:41.522101+0000","last_peered":"2023-10-11T04:28:41.522101+0000","last_clean":"2023-10-11T04:28:41.522101+0000","last_became_active":"2023-10-11T04:28:39.536769+0000","last_became_peered":"2023-10-11T04:28:39.536769+0000","last_unstale":"2023-10-11T04:28:41.522101+0000","last_undegraded":"2023-10-11T04:28:41.522101+0000","last_fullsized":"2023-10-11T04:28:41.522101+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:38.463092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:38.463092+0000","last_clean_scrub_stamp":"2023-10-11T04:28:38.463092+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-11T04:28:45.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-11T04:28:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:45.222+0000 7f71d606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:45.227+0000 7f71d606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:45.410 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-10-11T04:28:45.421 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-11T04:28:43.453972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59922,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.942511"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.522103+0000","last_change":"2023-10-11T04:28:40.594324+0000","last_active":"2023-10-11T04:28:41.522103+0000","last_peered":"2023-10-11T04:28:41.522103+0000","last_clean":"2023-10-11T04:28:41.522103+0000","last_became_active":"2023-10-11T04:28:40.592168+0000","last_became_peered":"2023-10-11T04:28:40.592168+0000","last_unstale":"2023-10-11T04:28:41.522103+0000","last_undegraded":"2023-10-11T04:28:41.522103+0000","last_fullsized":"2023-10-11T04:28:41.522103+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.523765+0000","last_change":"2023-10-11T04:28:40.592383+0000","last_active":"2023-10-11T04:28:41.523765+0000","last_peered":"2023-10-11T04:28:41.523765+0000","last_clean":"2023-10-11T04:28:41.523765+0000","last_became_active":"2023-10-11T04:28:40.590469+0000","last_became_peered":"2023-10-11T04:28:40.590469+0000","last_unstale":"2023-10-11T04:28:41.523765+0000","last_undegraded":"2023-10-11T04:28:41.523765+0000","last_fullsized":"2023-10-11T04:28:41.523765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.523199+0000","last_change":"2023-10-11T04:28:40.573925+0000","last_active":"2023-10-11T04:28:41.523199+0000","last_peered":"2023-10-11T04:28:41.523199+0000","last_clean":"2023-10-11T04:28:41.523199+0000","last_became_active":"2023-10-11T04:28:40.561298+0000","last_became_peered":"2023-10-11T04:28:40.561298+0000","last_unstale":"2023-10-11T04:28:41.523199+0000","last_undegraded":"2023-10-11T04:28:41.523199+0000","last_fullsized":"2023-10-11T04:28:41.523199+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.511155+0000","last_change":"2023-10-11T04:28:40.606076+0000","last_active":"2023-10-11T04:28:41.511155+0000","last_peered":"2023-10-11T04:28:41.511155+0000","last_clean":"2023-10-11T04:28:41.511155+0000","last_became_active":"2023-10-11T04:28:40.602843+0000","last_became_peered":"2023-10-11T04:28:40.602843+0000","last_unstale":"2023-10-11T04:28:41.511155+0000","last_undegraded":"2023-10-11T04:28:41.511155+0000","last_fullsized":"2023-10-11T04:28:41.511155+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.527648+0000","last_change":"2023-10-11T04:28:40.584585+0000","last_active":"2023-10-11T04:28:41.527648+0000","last_peered":"2023-10-11T04:28:41.527648+0000","last_clean":"2023-10-11T04:28:41.527648+0000","last_became_active":"2023-10-11T04:28:40.582698+0000","last_became_peered":"2023-10-11T04:28:40.582698+0000","last_unstale":"2023-10-11T04:28:41.527648+0000","last_undegraded":"2023-10-11T04:28:41.527648+0000","last_fullsized":"2023-10-11T04:28:41.527648+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.486328+0000","last_change":"2023-10-11T04:28:40.557577+0000","last_active":"2023-10-11T04:28:41.486328+0000","last_peered":"2023-10-11T04:28:41.486328+0000","last_clean":"2023-10-11T04:28:41.486328+0000","last_became_active":"2023-10-11T04:28:40.555762+0000","last_became_peered":"2023-10-11T04:28:40.555762+0000","last_unstale":"2023-10-11T04:28:41.486328+0000","last_undegraded":"2023-10-11T04:28:41.486328+0000","last_fullsized":"2023-10-11T04:28:41.486328+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.486113+0000","last_change":"2023-10-11T04:28:40.551449+0000","last_active":"2023-10-11T04:28:41.486113+0000","last_peered":"2023-10-11T04:28:41.486113+0000","last_clean":"2023-10-11T04:28:41.486113+0000","last_became_active":"2023-10-11T04:28:40.549180+0000","last_became_peered":"2023-10-11T04:28:40.549180+0000","last_unstale":"2023-10-11T04:28:41.486113+0000","last_undegraded":"2023-10-11T04:28:41.486113+0000","last_fullsized":"2023-10-11T04:28:41.486113+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.516854+0000","last_change":"2023-10-11T04:28:40.559705+0000","last_active":"2023-10-11T04:28:41.516854+0000","last_peered":"2023-10-11T04:28:41.516854+0000","last_clean":"2023-10-11T04:28:41.516854+0000","last_became_active":"2023-10-11T04:28:40.556648+0000","last_became_peered":"2023-10-11T04:28:40.556648+0000","last_unstale":"2023-10-11T04:28:41.516854+0000","last_undegraded":"2023-10-11T04:28:41.516854+0000","last_fullsized":"2023-10-11T04:28:41.516854+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:39.468684+0000","last_clean_scrub_stamp":"2023-10-11T04:28:39.468684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-11T04:28:41.522101+0000","last_change":"2023-10-11T04:28:39.546833+0000","last_active":"2023-10-11T04:28:41.522101+0000","last_peered":"2023-10-11T04:28:41.522101+0000","last_clean":"2023-10-11T04:28:41.522101+0000","last_became_active":"2023-10-11T04:28:39.536769+0000","last_became_peered":"2023-10-11T04:28:39.536769+0000","last_unstale":"2023-10-11T04:28:41.522101+0000","last_undegraded":"2023-10-11T04:28:41.522101+0000","last_fullsized":"2023-10-11T04:28:41.522101+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-11T04:28:38.463092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-11T04:28:38.463092+0000","last_clean_scrub_stamp":"2023-10-11T04:28:38.463092+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-11T04:28:45.422 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-11T04:28:45.423 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-11T04:28:45.423 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-11T04:28:45.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-11T04:28:45.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:45.531+0000 7f124e71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:28:45.535+0000 7f124e71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:28:45.753 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:45.754 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-11T04:28:45.765 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-11T04:28:45.766 INFO:teuthology.run_tasks:Running task cbt... 2023-10-11T04:28:45.779 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi040.front.sepia.ceph.com', 'osds': ['ubuntu@smithi040.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi040.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi040.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi040.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-10-11T04:28:45.780 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-10-11T04:28:45.799 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:28:45.799 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-11T04:28:45.851 DEBUG:teuthology.orchestra.run.smithi040:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-10-11T04:28:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-10-11T04:28:46.358 INFO:tasks.cbt:Installing collectl 2023-10-11T04:28:46.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-10-11T04:28:46.467 INFO:teuthology.orchestra.run.smithi040.stderr:--2023-10-11 04:28:46-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-10-11T04:28:46.608 INFO:teuthology.orchestra.run.smithi040.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2023-10-11T04:28:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-10-11T04:28:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-11T04:28:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlJiR-MEOx0vn90cBTiUaJf0Apz5ZCt8xCBeWXwQ7-dbRcZDXMq6exB2LCeiEEufLDKa77C05nvl1FuPVQYPS7BUZ3Zw%3D%3D&use_mirror=newcontinuum&r= [following] 2023-10-11T04:28:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:--2023-10-11 04:28:46-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlJiR-MEOx0vn90cBTiUaJf0Apz5ZCt8xCBeWXwQ7-dbRcZDXMq6exB2LCeiEEufLDKa77C05nvl1FuPVQYPS7BUZ3Zw%3D%3D&use_mirror=newcontinuum&r= 2023-10-11T04:28:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-10-11T04:28:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-10-11T04:28:47.214 INFO:teuthology.orchestra.run.smithi040.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-11T04:28:47.214 INFO:teuthology.orchestra.run.smithi040.stderr:Location: https://newcontinuum.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-10-11T04:28:47.215 INFO:teuthology.orchestra.run.smithi040.stderr:--2023-10-11 04:28:47-- https://newcontinuum.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-10-11T04:28:47.215 INFO:teuthology.orchestra.run.smithi040.stderr:Resolving newcontinuum.dl.sourceforge.net (newcontinuum.dl.sourceforge.net)... 64.79.96.4, 2607:ff50:0:11::32 2023-10-11T04:28:47.255 INFO:teuthology.orchestra.run.smithi040.stderr:Connecting to newcontinuum.dl.sourceforge.net (newcontinuum.dl.sourceforge.net)|64.79.96.4|:443... connected. 2023-10-11T04:28:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-11T04:28:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:Length: 620693 (606K) [application/x-gzip] 2023-10-11T04:28:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:Saving to: ‘collectl.tar.gz’ 2023-10-11T04:28:48.795 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:48.835 INFO:teuthology.orchestra.run.smithi040.stderr: 0K .......... .......... .......... .......... .......... 8% 625K 1s 2023-10-11T04:28:48.875 INFO:teuthology.orchestra.run.smithi040.stderr: 50K .......... .......... .......... .......... .......... 16% 1.22M 1s 2023-10-11T04:28:49.501 INFO:teuthology.orchestra.run.smithi040.stderr: 100K .......... .......... .......... .......... .......... 24% 80.0K 2s 2023-10-11T04:28:49.542 INFO:teuthology.orchestra.run.smithi040.stderr: 150K .......... .......... .......... .......... .......... 32% 1.17M 2s 2023-10-11T04:28:49.585 INFO:teuthology.orchestra.run.smithi040.stderr: 200K .......... .......... .......... .......... .......... 41% 1.15M 1s 2023-10-11T04:28:49.587 INFO:teuthology.orchestra.run.smithi040.stderr: 250K .......... .......... .......... .......... .......... 49% 21.1M 1s 2023-10-11T04:28:49.623 INFO:teuthology.orchestra.run.smithi040.stderr: 300K .......... .......... .......... .......... .......... 57% 1.38M 1s 2023-10-11T04:28:49.635 INFO:teuthology.orchestra.run.smithi040.stderr: 350K .......... .......... .......... .......... .......... 65% 5.70M 0s 2023-10-11T04:28:49.640 INFO:teuthology.orchestra.run.smithi040.stderr: 400K .......... .......... .......... .......... .......... 74% 5.40M 0s 2023-10-11T04:28:49.650 INFO:teuthology.orchestra.run.smithi040.stderr: 450K .......... .......... .......... .......... .......... 82% 4.99M 0s 2023-10-11T04:28:49.654 INFO:teuthology.orchestra.run.smithi040.stderr: 500K .......... .......... .......... .......... .......... 90% 12.5M 0s 2023-10-11T04:28:50.018 INFO:teuthology.orchestra.run.smithi040.stderr: 550K .......... .......... .......... .......... .......... 98% 138K 0s 2023-10-11T04:28:50.018 INFO:teuthology.orchestra.run.smithi040.stderr: 600K ...... 100% 150M=1.3s 2023-10-11T04:28:50.018 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:50.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11 04:28:50 (480 KB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-10-11T04:28:50.019 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:28:50.055 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/ 2023-10-11T04:28:50.055 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/collectl.conf 2023-10-11T04:28:50.055 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/ 2023-10-11T04:28:50.056 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/SlowProc.html 2023-10-11T04:28:50.056 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-10-11T04:28:50.056 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/WhySummary.html 2023-10-11T04:28:50.057 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Socket.html 2023-10-11T04:28:50.057 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/colmux.html 2023-10-11T04:28:50.057 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Tutorial.html 2023-10-11T04:28:50.058 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Gexpr.html 2023-10-11T04:28:50.059 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Import.html 2023-10-11T04:28:50.059 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/style.css 2023-10-11T04:28:50.059 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-10-11T04:28:50.060 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-10-11T04:28:50.060 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/FileNaming.html 2023-10-11T04:28:50.060 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/TheMath.html 2023-10-11T04:28:50.061 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Environmental.html 2023-10-11T04:28:50.061 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Process.html 2023-10-11T04:28:50.061 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-10-11T04:28:50.061 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/HiResTime.html 2023-10-11T04:28:50.062 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-10-11T04:28:50.062 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/slub.jpg 2023-10-11T04:28:50.069 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Lustre.html 2023-10-11T04:28:50.069 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Documentation.html 2023-10-11T04:28:50.070 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Inodes.html 2023-10-11T04:28:50.070 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Graphite.html 2023-10-11T04:28:50.070 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/InputFiles.html 2023-10-11T04:28:50.070 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Export.html 2023-10-11T04:28:50.070 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Interrupts.html 2023-10-11T04:28:50.071 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Startup.html 2023-10-11T04:28:50.071 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Performance.html 2023-10-11T04:28:50.071 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Messages.html 2023-10-11T04:28:50.071 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Logging.html 2023-10-11T04:28:50.071 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Data-detail.html 2023-10-11T04:28:50.072 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/CPUs.html 2023-10-11T04:28:50.072 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Matrix.html 2023-10-11T04:28:50.072 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Data-brief.html 2023-10-11T04:28:50.072 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-10-11T04:28:50.072 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Misc.html 2023-10-11T04:28:50.073 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Examples.html 2023-10-11T04:28:50.073 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/OpenStack.html 2023-10-11T04:28:50.073 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-10-11T04:28:50.073 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-10-11T04:28:50.073 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-10-11T04:28:50.074 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-10-11T04:28:50.074 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Data.html 2023-10-11T04:28:50.074 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Exceptions.html 2023-10-11T04:28:50.075 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-10-11T04:28:50.075 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-10-11T04:28:50.075 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-10-11T04:28:50.075 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-10-11T04:28:50.075 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Playback.html 2023-10-11T04:28:50.076 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Hello.html 2023-10-11T04:28:50.076 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Sockets.html 2023-10-11T04:28:50.077 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Network.html 2023-10-11T04:28:50.077 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Infiniband.html 2023-10-11T04:28:50.077 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Memory.html 2023-10-11T04:28:50.077 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-10-11T04:28:50.077 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/Disks.html 2023-10-11T04:28:50.078 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-10-11T04:28:50.078 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/service/ 2023-10-11T04:28:50.078 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/service/collectl.service 2023-10-11T04:28:50.078 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/initd/ 2023-10-11T04:28:50.079 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/initd/collectl-suse 2023-10-11T04:28:50.079 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/initd/collectl-debian 2023-10-11T04:28:50.079 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/initd/collectl-generic 2023-10-11T04:28:50.079 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/initd/collectl 2023-10-11T04:28:50.080 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/misc.ph 2023-10-11T04:28:50.080 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/gexpr.ph 2023-10-11T04:28:50.080 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/statsd.ph 2023-10-11T04:28:50.081 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/envrules.std 2023-10-11T04:28:50.081 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/hello.ph 2023-10-11T04:28:50.081 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/vmsum.ph 2023-10-11T04:28:50.082 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/proctree.ph 2023-10-11T04:28:50.082 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/README-WINDOWS 2023-10-11T04:28:50.083 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/formatit.ph 2023-10-11T04:28:50.083 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/colmux 2023-10-11T04:28:50.083 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/INSTALL 2023-10-11T04:28:50.084 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/graphite.ph 2023-10-11T04:28:50.084 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/RELEASE-collectl 2023-10-11T04:28:50.084 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/vmstat.ph 2023-10-11T04:28:50.085 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/README 2023-10-11T04:28:50.085 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/man1/ 2023-10-11T04:28:50.085 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/man1/collectl.1 2023-10-11T04:28:50.086 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/man1/colmux.1 2023-10-11T04:28:50.086 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/lexpr.ph 2023-10-11T04:28:50.087 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/client.pl 2023-10-11T04:28:50.087 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/collectl 2023-10-11T04:28:50.087 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/COPYING 2023-10-11T04:28:50.088 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/vnet.ph 2023-10-11T04:28:50.088 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/GPL 2023-10-11T04:28:50.088 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/UNINSTALL 2023-10-11T04:28:50.088 INFO:teuthology.orchestra.run.smithi040.stdout:collectl-4.3.1/ARTISTIC 2023-10-11T04:28:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:./INSTALL: line 90: chkconfig: command not found 2023-10-11T04:28:50.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-10-11T04:28:50.745 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:35 ago on Wed 11 Oct 2023 04:25:15 AM UTC. 2023-10-11T04:28:50.837 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-10-11T04:28:50.840 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-10-11T04:28:50.896 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-10-11T04:28:50.897 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:50.897 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-10-11T04:28:50.897 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:50.897 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-10-11T04:28:50.897 INFO:teuthology.orchestra.run.smithi040.stdout: librbd-devel x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 23 k 2023-10-11T04:28:50.898 INFO:teuthology.orchestra.run.smithi040.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-10-11T04:28:50.898 INFO:teuthology.orchestra.run.smithi040.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-10-11T04:28:50.898 INFO:teuthology.orchestra.run.smithi040.stdout: perf x86_64 5.14.0-373.el9 CentOS-AppStream 7.2 M 2023-10-11T04:28:50.898 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-10-11T04:28:50.898 INFO:teuthology.orchestra.run.smithi040.stdout: libradospp-devel x86_64 2:18.0.0-6684.g9587ffdf.el9 ceph 30 k 2023-10-11T04:28:50.899 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:50.899 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-10-11T04:28:50.899 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-10-11T04:28:50.899 INFO:teuthology.orchestra.run.smithi040.stdout:Install 5 Packages 2023-10-11T04:28:50.899 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:50.900 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 7.4 M 2023-10-11T04:28:50.900 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 11 M 2023-10-11T04:28:50.901 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-10-11T04:28:51.277 INFO:teuthology.orchestra.run.smithi040.stdout:(1/5): perf-5.14.0-373.el9.x86_64.rpm 22 MB/s | 7.2 MB 00:00 2023-10-11T04:28:51.295 INFO:teuthology.orchestra.run.smithi040.stdout:(2/5): librbd-devel-18.0.0-6684.g9587ffdf.el9.x 67 kB/s | 23 kB 00:00 2023-10-11T04:28:51.436 INFO:teuthology.orchestra.run.smithi040.stdout:(3/5): libradospp-devel-18.0.0-6684.g9587ffdf.e 62 kB/s | 30 kB 00:00 2023-10-11T04:28:51.520 INFO:teuthology.orchestra.run.smithi040.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 58 kB/s | 13 kB 00:00 2023-10-11T04:28:51.661 INFO:teuthology.orchestra.run.smithi040.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 309 kB/s | 118 kB 00:00 2023-10-11T04:28:51.665 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-10-11T04:28:51.665 INFO:teuthology.orchestra.run.smithi040.stdout:Total 9.7 MB/s | 7.4 MB 00:00 2023-10-11T04:28:51.665 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-10-11T04:28:51.688 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-10-11T04:28:51.688 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-10-11T04:28:51.857 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-10-11T04:28:51.858 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-10-11T04:28:52.162 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-10-11T04:28:52.230 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-10-11T04:28:52.269 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-10-11T04:28:52.300 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradospp-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 3/5 2023-10-11T04:28:52.646 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librbd-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 4/5 2023-10-11T04:28:52.784 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : perf-5.14.0-373.el9.x86_64 5/5 2023-10-11T04:28:53.797 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: perf-5.14.0-373.el9.x86_64 5/5 2023-10-11T04:28:53.798 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradospp-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 1/5 2023-10-11T04:28:53.798 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 2/5 2023-10-11T04:28:53.798 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : perf-5.14.0-373.el9.x86_64 3/5 2023-10-11T04:28:53.799 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-10-11T04:28:54.175 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-10-11T04:28:54.175 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:54.175 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-10-11T04:28:54.176 INFO:teuthology.orchestra.run.smithi040.stdout: libradospp-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:28:54.176 INFO:teuthology.orchestra.run.smithi040.stdout: librbd-devel-2:18.0.0-6684.g9587ffdf.el9.x86_64 2023-10-11T04:28:54.176 INFO:teuthology.orchestra.run.smithi040.stdout: pdsh-2.34-7.el9.x86_64 2023-10-11T04:28:54.176 INFO:teuthology.orchestra.run.smithi040.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-10-11T04:28:54.176 INFO:teuthology.orchestra.run.smithi040.stdout: perf-5.14.0-373.el9.x86_64 2023-10-11T04:28:54.176 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:54.177 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-10-11T04:28:54.557 INFO:tasks.cbt:benchmark: librbdfio 2023-10-11T04:28:54.558 DEBUG:teuthology.orchestra.run.smithi040:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2023-10-11T04:28:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2023-10-11T04:28:56.687 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest/fio && ./configure && make 2023-10-11T04:28:57.302 INFO:teuthology.orchestra.run.smithi040.stdout:Operating system Linux 2023-10-11T04:28:57.303 INFO:teuthology.orchestra.run.smithi040.stdout:CPU x86_64 2023-10-11T04:28:57.303 INFO:teuthology.orchestra.run.smithi040.stdout:Big endian no 2023-10-11T04:28:57.303 INFO:teuthology.orchestra.run.smithi040.stdout:Compiler gcc 2023-10-11T04:28:57.304 INFO:teuthology.orchestra.run.smithi040.stdout:Cross compile no 2023-10-11T04:28:57.304 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:28:57.304 INFO:teuthology.orchestra.run.smithi040.stdout:Static build no 2023-10-11T04:28:57.425 INFO:teuthology.orchestra.run.smithi040.stdout:Wordsize 64 2023-10-11T04:28:57.432 INFO:teuthology.orchestra.run.smithi040.stdout:zlib no 2023-10-11T04:28:57.484 INFO:teuthology.orchestra.run.smithi040.stdout:fcntl(F_FULLFSYNC) no 2023-10-11T04:28:57.498 INFO:teuthology.orchestra.run.smithi040.stdout:Linux AIO support no 2023-10-11T04:28:57.499 INFO:teuthology.orchestra.run.smithi040.stdout:Linux AIO support rw flags no 2023-10-11T04:28:57.563 INFO:teuthology.orchestra.run.smithi040.stdout:POSIX AIO support yes 2023-10-11T04:28:57.564 INFO:teuthology.orchestra.run.smithi040.stdout:POSIX AIO support needs -lrt no 2023-10-11T04:28:57.595 INFO:teuthology.orchestra.run.smithi040.stdout:POSIX AIO fsync yes 2023-10-11T04:28:57.625 INFO:teuthology.orchestra.run.smithi040.stdout:POSIX pshared support yes 2023-10-11T04:28:57.682 INFO:teuthology.orchestra.run.smithi040.stdout:pthread_condattr_setclock() yes 2023-10-11T04:28:57.720 INFO:teuthology.orchestra.run.smithi040.stdout:pthread_sigmask() yes 2023-10-11T04:28:57.756 INFO:teuthology.orchestra.run.smithi040.stdout:pthread_getaffinity_np() yes 2023-10-11T04:28:57.766 INFO:teuthology.orchestra.run.smithi040.stdout:Solaris AIO support no 2023-10-11T04:28:57.817 INFO:teuthology.orchestra.run.smithi040.stdout:__sync_fetch_and_add yes 2023-10-11T04:28:57.857 INFO:teuthology.orchestra.run.smithi040.stdout:__sync_synchronize yes 2023-10-11T04:28:57.887 INFO:teuthology.orchestra.run.smithi040.stdout:__sync_val_compare_and_swap yes 2023-10-11T04:28:57.894 INFO:teuthology.orchestra.run.smithi040.stdout:libverbs no 2023-10-11T04:28:57.933 INFO:teuthology.orchestra.run.smithi040.stdout:rdmacm no 2023-10-11T04:28:57.941 INFO:teuthology.orchestra.run.smithi040.stdout:librpma no 2023-10-11T04:28:57.950 INFO:teuthology.orchestra.run.smithi040.stdout:libprotobuf_c no 2023-10-11T04:28:57.982 INFO:teuthology.orchestra.run.smithi040.stdout:asprintf() yes 2023-10-11T04:28:58.013 INFO:teuthology.orchestra.run.smithi040.stdout:vasprintf() yes 2023-10-11T04:28:58.044 INFO:teuthology.orchestra.run.smithi040.stdout:Linux fallocate yes 2023-10-11T04:28:58.077 INFO:teuthology.orchestra.run.smithi040.stdout:POSIX fadvise yes 2023-10-11T04:28:58.109 INFO:teuthology.orchestra.run.smithi040.stdout:POSIX fallocate yes 2023-10-11T04:28:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:sched_setaffinity(3 arg) yes 2023-10-11T04:28:58.139 INFO:teuthology.orchestra.run.smithi040.stdout:sched_setaffinity(2 arg) no 2023-10-11T04:28:58.171 INFO:teuthology.orchestra.run.smithi040.stdout:clock_gettime yes 2023-10-11T04:28:58.203 INFO:teuthology.orchestra.run.smithi040.stdout:CLOCK_MONOTONIC yes 2023-10-11T04:28:58.239 INFO:teuthology.orchestra.run.smithi040.stdout:clockid_t yes 2023-10-11T04:28:58.289 INFO:teuthology.orchestra.run.smithi040.stdout:gettimeofday yes 2023-10-11T04:28:58.322 INFO:teuthology.orchestra.run.smithi040.stdout:fdatasync yes 2023-10-11T04:28:58.352 INFO:teuthology.orchestra.run.smithi040.stdout:pipe() yes 2023-10-11T04:28:58.384 INFO:teuthology.orchestra.run.smithi040.stdout:pipe2() yes 2023-10-11T04:28:58.414 INFO:teuthology.orchestra.run.smithi040.stdout:pread() yes 2023-10-11T04:28:58.505 INFO:teuthology.orchestra.run.smithi040.stdout:sync_file_range yes 2023-10-11T04:28:58.512 INFO:teuthology.orchestra.run.smithi040.stdout:ASharedMemory_create no 2023-10-11T04:28:58.533 INFO:teuthology.orchestra.run.smithi040.stdout:EXT4 move extent yes 2023-10-11T04:28:58.564 INFO:teuthology.orchestra.run.smithi040.stdout:Linux splice(2) yes 2023-10-11T04:28:58.572 INFO:teuthology.orchestra.run.smithi040.stdout:libnuma no 2023-10-11T04:28:58.602 INFO:teuthology.orchestra.run.smithi040.stdout:strsep yes 2023-10-11T04:28:58.632 INFO:teuthology.orchestra.run.smithi040.stdout:strcasestr yes 2023-10-11T04:28:58.646 INFO:teuthology.orchestra.run.smithi040.stdout:strlcat no 2023-10-11T04:28:58.680 INFO:teuthology.orchestra.run.smithi040.stdout:getopt_long_only() yes 2023-10-11T04:28:58.754 INFO:teuthology.orchestra.run.smithi040.stdout:inet_aton yes 2023-10-11T04:28:58.787 INFO:teuthology.orchestra.run.smithi040.stdout:socklen_t yes 2023-10-11T04:28:58.820 INFO:teuthology.orchestra.run.smithi040.stdout:__thread yes 2023-10-11T04:28:58.850 INFO:teuthology.orchestra.run.smithi040.stdout:RUSAGE_THREAD yes 2023-10-11T04:28:58.880 INFO:teuthology.orchestra.run.smithi040.stdout:SCHED_IDLE yes 2023-10-11T04:28:58.915 INFO:teuthology.orchestra.run.smithi040.stdout:TCP_NODELAY yes 2023-10-11T04:28:58.949 INFO:teuthology.orchestra.run.smithi040.stdout:Net engine window_size yes 2023-10-11T04:28:58.984 INFO:teuthology.orchestra.run.smithi040.stdout:TCP_MAXSEG yes 2023-10-11T04:28:59.015 INFO:teuthology.orchestra.run.smithi040.stdout:RLIMIT_MEMLOCK yes 2023-10-11T04:28:59.049 INFO:teuthology.orchestra.run.smithi040.stdout:pwritev/preadv yes 2023-10-11T04:28:59.082 INFO:teuthology.orchestra.run.smithi040.stdout:pwritev2/preadv2 yes 2023-10-11T04:28:59.119 INFO:teuthology.orchestra.run.smithi040.stdout:IPv6 helpers yes 2023-10-11T04:28:59.136 INFO:teuthology.orchestra.run.smithi040.stdout:http engine no 2023-10-11T04:28:59.231 INFO:teuthology.orchestra.run.smithi040.stdout:Rados engine yes 2023-10-11T04:28:59.337 INFO:teuthology.orchestra.run.smithi040.stdout:Rados Block Device engine yes 2023-10-11T04:28:59.448 INFO:teuthology.orchestra.run.smithi040.stdout:rbd_poll yes 2023-10-11T04:28:59.556 INFO:teuthology.orchestra.run.smithi040.stdout:rbd_invalidate_cache yes 2023-10-11T04:28:59.587 INFO:teuthology.orchestra.run.smithi040.stdout:setvbuf yes 2023-10-11T04:28:59.595 INFO:teuthology.orchestra.run.smithi040.stdout:Gluster API engine no 2023-10-11T04:28:59.610 INFO:teuthology.orchestra.run.smithi040.stdout:s390_z196_facilities no 2023-10-11T04:28:59.610 INFO:teuthology.orchestra.run.smithi040.stdout:HDFS engine no 2023-10-11T04:28:59.662 INFO:teuthology.orchestra.run.smithi040.stdout:MTD yes 2023-10-11T04:28:59.670 INFO:teuthology.orchestra.run.smithi040.stdout:libpmem no 2023-10-11T04:28:59.670 INFO:teuthology.orchestra.run.smithi040.stdout:libpmem1_5 no 2023-10-11T04:28:59.676 INFO:teuthology.orchestra.run.smithi040.stdout:libpmem2 no 2023-10-11T04:28:59.677 INFO:teuthology.orchestra.run.smithi040.stdout:PMDK dev-dax engine no 2023-10-11T04:28:59.677 INFO:teuthology.orchestra.run.smithi040.stdout:PMDK libpmem engine no 2023-10-11T04:28:59.684 INFO:teuthology.orchestra.run.smithi040.stdout:DDN's Infinite Memory Engine no 2023-10-11T04:28:59.685 INFO:teuthology.orchestra.run.smithi040.stdout:iscsi engine no 2023-10-11T04:28:59.685 INFO:teuthology.orchestra.run.smithi040.stdout:NBD engine no 2023-10-11T04:28:59.693 INFO:teuthology.orchestra.run.smithi040.stdout:DAOS File System (dfs) Engine no 2023-10-11T04:28:59.702 INFO:teuthology.orchestra.run.smithi040.stdout:NFS engine no 2023-10-11T04:28:59.703 INFO:teuthology.orchestra.run.smithi040.stdout:lex/yacc for arithmetic no 2023-10-11T04:28:59.745 INFO:teuthology.orchestra.run.smithi040.stdout:getmntent yes 2023-10-11T04:28:59.768 INFO:teuthology.orchestra.run.smithi040.stdout:getmntinfo no 2023-10-11T04:28:59.834 INFO:teuthology.orchestra.run.smithi040.stdout:Static Assert yes 2023-10-11T04:28:59.861 INFO:teuthology.orchestra.run.smithi040.stdout:bool yes 2023-10-11T04:28:59.896 INFO:teuthology.orchestra.run.smithi040.stdout:strndup yes 2023-10-11T04:28:59.903 INFO:teuthology.orchestra.run.smithi040.stdout:Valgrind headers no 2023-10-11T04:28:59.943 INFO:teuthology.orchestra.run.smithi040.stdout:Zoned block device support yes 2023-10-11T04:28:59.972 INFO:teuthology.orchestra.run.smithi040.stdout:Zoned block device capacity yes 2023-10-11T04:28:59.979 INFO:teuthology.orchestra.run.smithi040.stdout:libzbc engine no 2023-10-11T04:29:00.009 INFO:teuthology.orchestra.run.smithi040.stdout:NVMe uring command support yes 2023-10-11T04:29:00.023 INFO:teuthology.orchestra.run.smithi040.stdout:xnvme engine no 2023-10-11T04:29:00.038 INFO:teuthology.orchestra.run.smithi040.stdout:libblkio engine no 2023-10-11T04:29:00.039 INFO:teuthology.orchestra.run.smithi040.stdout:march_armv8_a_crc_crypto no 2023-10-11T04:29:00.039 INFO:teuthology.orchestra.run.smithi040.stdout:cuda no 2023-10-11T04:29:00.039 INFO:teuthology.orchestra.run.smithi040.stdout:libcufile no 2023-10-11T04:29:00.065 INFO:teuthology.orchestra.run.smithi040.stdout:Build march=native yes 2023-10-11T04:29:00.073 INFO:teuthology.orchestra.run.smithi040.stdout:CUnit no 2023-10-11T04:29:00.102 INFO:teuthology.orchestra.run.smithi040.stdout:__kernel_rwf_t yes 2023-10-11T04:29:00.130 INFO:teuthology.orchestra.run.smithi040.stdout:-Wimplicit-fallthrough=2 yes 2023-10-11T04:29:00.161 INFO:teuthology.orchestra.run.smithi040.stdout:-Wno-stringop-truncation yes 2023-10-11T04:29:00.192 INFO:teuthology.orchestra.run.smithi040.stdout:MADV_HUGEPAGE yes 2023-10-11T04:29:00.222 INFO:teuthology.orchestra.run.smithi040.stdout:gettid yes 2023-10-11T04:29:00.257 INFO:teuthology.orchestra.run.smithi040.stdout:statx(2)/libc yes 2023-10-11T04:29:00.292 INFO:teuthology.orchestra.run.smithi040.stdout:statx(2)/syscall yes 2023-10-11T04:29:00.294 INFO:teuthology.orchestra.run.smithi040.stdout:Windows PDB generation no 2023-10-11T04:29:00.325 INFO:teuthology.orchestra.run.smithi040.stdout:timerfd_create yes 2023-10-11T04:29:00.327 INFO:teuthology.orchestra.run.smithi040.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2023-10-11T04:29:00.328 INFO:teuthology.orchestra.run.smithi040.stdout:Lib-based ioengines dynamic no 2023-10-11T04:29:00.391 INFO:teuthology.orchestra.run.smithi040.stdout:TCMalloc support yes 2023-10-11T04:29:00.392 INFO:teuthology.orchestra.run.smithi040.stdout:seed_buckets 4 2023-10-11T04:29:00.483 INFO:teuthology.orchestra.run.smithi040.stderr:FIO_VERSION = fio-3.35-137-g16b9f2 2023-10-11T04:29:00.520 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc16.o 2023-10-11T04:29:00.613 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc32.o 2023-10-11T04:29:00.650 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc32c-arm64.o 2023-10-11T04:29:00.728 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc32c-intel.o 2023-10-11T04:29:00.772 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc32c.o 2023-10-11T04:29:00.812 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc64.o 2023-10-11T04:29:00.851 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crc7.o 2023-10-11T04:29:00.885 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/crct10dif_common.o 2023-10-11T04:29:00.919 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/fnv.o 2023-10-11T04:29:00.960 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/md5.o 2023-10-11T04:29:01.066 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/murmur3.o 2023-10-11T04:29:01.110 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/sha1.o 2023-10-11T04:29:01.297 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/sha256.o 2023-10-11T04:29:01.608 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/sha3.o 2023-10-11T04:29:01.759 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/sha512.o 2023-10-11T04:29:01.878 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/test.o 2023-10-11T04:29:02.045 INFO:teuthology.orchestra.run.smithi040.stdout: CC crc/xxhash.o 2023-10-11T04:29:02.144 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/axmap.o 2023-10-11T04:29:02.271 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/bloom.o 2023-10-11T04:29:02.356 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/flist_sort.o 2023-10-11T04:29:02.429 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/gauss.o 2023-10-11T04:29:02.521 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/getrusage.o 2023-10-11T04:29:02.557 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/hweight.o 2023-10-11T04:29:02.595 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/ieee754.o 2023-10-11T04:29:02.645 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/lfsr.o 2023-10-11T04:29:02.748 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/memalign.o 2023-10-11T04:29:02.791 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/memcpy.o 2023-10-11T04:29:02.940 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/mountcheck.o 2023-10-11T04:29:02.980 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/num2str.o 2023-10-11T04:29:03.047 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/output_buffer.o 2023-10-11T04:29:03.095 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/pattern.o 2023-10-11T04:29:03.246 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/prio_tree.o 2023-10-11T04:29:03.371 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/rand.o 2023-10-11T04:29:03.566 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/rbtree.o 2023-10-11T04:29:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/strntol.o 2023-10-11T04:29:03.740 INFO:teuthology.orchestra.run.smithi040.stdout: CC lib/zipf.o 2023-10-11T04:29:03.863 INFO:teuthology.orchestra.run.smithi040.stdout: CC gettime.o 2023-10-11T04:29:04.166 INFO:teuthology.orchestra.run.smithi040.stdout: CC ioengines.o 2023-10-11T04:29:04.435 INFO:teuthology.orchestra.run.smithi040.stdout: CC init.o 2023-10-11T04:29:05.239 INFO:teuthology.orchestra.run.smithi040.stdout: CC stat.o 2023-10-11T04:29:07.216 INFO:teuthology.orchestra.run.smithi040.stdout: CC log.o 2023-10-11T04:29:07.340 INFO:teuthology.orchestra.run.smithi040.stdout: CC time.o 2023-10-11T04:29:07.465 INFO:teuthology.orchestra.run.smithi040.stdout: CC filesetup.o 2023-10-11T04:29:08.178 INFO:teuthology.orchestra.run.smithi040.stdout: CC eta.o 2023-10-11T04:29:08.509 INFO:teuthology.orchestra.run.smithi040.stdout: CC verify.o 2023-10-11T04:29:09.216 INFO:teuthology.orchestra.run.smithi040.stdout: CC memory.o 2023-10-11T04:29:09.396 INFO:teuthology.orchestra.run.smithi040.stdout: CC io_u.o 2023-10-11T04:29:10.176 INFO:teuthology.orchestra.run.smithi040.stdout: CC parse.o 2023-10-11T04:29:10.639 INFO:teuthology.orchestra.run.smithi040.stdout: CC fio_sem.o 2023-10-11T04:29:10.744 INFO:teuthology.orchestra.run.smithi040.stdout: CC rwlock.o 2023-10-11T04:29:10.830 INFO:teuthology.orchestra.run.smithi040.stdout: CC pshared.o 2023-10-11T04:29:10.896 INFO:teuthology.orchestra.run.smithi040.stdout: CC options.o 2023-10-11T04:29:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: CC smalloc.o 2023-10-11T04:29:11.898 INFO:teuthology.orchestra.run.smithi040.stdout: CC filehash.o 2023-10-11T04:29:12.054 INFO:teuthology.orchestra.run.smithi040.stdout: CC profile.o 2023-10-11T04:29:12.175 INFO:teuthology.orchestra.run.smithi040.stdout: CC debug.o 2023-10-11T04:29:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/cpu.o 2023-10-11T04:29:12.359 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/mmap.o 2023-10-11T04:29:12.534 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/sync.o 2023-10-11T04:29:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/null.o 2023-10-11T04:29:12.897 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/net.o 2023-10-11T04:29:13.330 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/ftruncate.o 2023-10-11T04:29:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/fileoperations.o 2023-10-11T04:29:13.570 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/exec.o 2023-10-11T04:29:13.725 INFO:teuthology.orchestra.run.smithi040.stdout: CC server.o 2023-10-11T04:29:14.486 INFO:teuthology.orchestra.run.smithi040.stdout: CC client.o 2023-10-11T04:29:15.038 INFO:teuthology.orchestra.run.smithi040.stdout: CC iolog.o 2023-10-11T04:29:15.490 INFO:teuthology.orchestra.run.smithi040.stdout: CC backend.o 2023-10-11T04:29:16.355 INFO:teuthology.orchestra.run.smithi040.stdout: CC libfio.o 2023-10-11T04:29:16.543 INFO:teuthology.orchestra.run.smithi040.stdout: CC flow.o 2023-10-11T04:29:16.661 INFO:teuthology.orchestra.run.smithi040.stdout: CC cconv.o 2023-10-11T04:29:17.020 INFO:teuthology.orchestra.run.smithi040.stdout: CC gettime-thread.o 2023-10-11T04:29:17.127 INFO:teuthology.orchestra.run.smithi040.stdout: CC helpers.o 2023-10-11T04:29:17.189 INFO:teuthology.orchestra.run.smithi040.stdout: CC json.o 2023-10-11T04:29:17.381 INFO:teuthology.orchestra.run.smithi040.stdout: CC idletime.o 2023-10-11T04:29:17.615 INFO:teuthology.orchestra.run.smithi040.stdout: CC td_error.o 2023-10-11T04:29:17.716 INFO:teuthology.orchestra.run.smithi040.stdout: CC profiles/tiobench.o 2023-10-11T04:29:17.817 INFO:teuthology.orchestra.run.smithi040.stdout: CC profiles/act.o 2023-10-11T04:29:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: CC io_u_queue.o 2023-10-11T04:29:18.030 INFO:teuthology.orchestra.run.smithi040.stdout: CC filelock.o 2023-10-11T04:29:18.158 INFO:teuthology.orchestra.run.smithi040.stdout: CC workqueue.o 2023-10-11T04:29:18.336 INFO:teuthology.orchestra.run.smithi040.stdout: CC rate-submit.o 2023-10-11T04:29:18.485 INFO:teuthology.orchestra.run.smithi040.stdout: CC optgroup.o 2023-10-11T04:29:18.533 INFO:teuthology.orchestra.run.smithi040.stdout: CC helper_thread.o 2023-10-11T04:29:18.712 INFO:teuthology.orchestra.run.smithi040.stdout: CC steadystate.o 2023-10-11T04:29:18.952 INFO:teuthology.orchestra.run.smithi040.stdout: CC zone-dist.o 2023-10-11T04:29:19.091 INFO:teuthology.orchestra.run.smithi040.stdout: CC zbd.o 2023-10-11T04:29:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: CC dedupe.o 2023-10-11T04:29:19.975 INFO:teuthology.orchestra.run.smithi040.stdout: CC fdp.o 2023-10-11T04:29:20.119 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/posixaio.o 2023-10-11T04:29:20.253 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/falloc.o 2023-10-11T04:29:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/e4defrag.o 2023-10-11T04:29:20.506 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/splice.o 2023-10-11T04:29:20.696 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/asprintf.o 2023-10-11T04:29:20.733 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/strlcat.o 2023-10-11T04:29:20.778 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/mtd.o 2023-10-11T04:29:20.945 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/libmtd.o 2023-10-11T04:29:21.342 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/libmtd_legacy.o 2023-10-11T04:29:21.477 INFO:teuthology.orchestra.run.smithi040.stdout: CC diskutil.o 2023-10-11T04:29:21.692 INFO:teuthology.orchestra.run.smithi040.stdout: CC fifo.o 2023-10-11T04:29:21.748 INFO:teuthology.orchestra.run.smithi040.stdout: CC blktrace.o 2023-10-11T04:29:22.092 INFO:teuthology.orchestra.run.smithi040.stdout: CC cgroup.o 2023-10-11T04:29:22.244 INFO:teuthology.orchestra.run.smithi040.stdout: CC trim.o 2023-10-11T04:29:22.362 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/sg.o 2023-10-11T04:29:22.760 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/linux-dev-lookup.o 2023-10-11T04:29:22.819 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/io_uring.o 2023-10-11T04:29:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/nvme.o 2023-10-11T04:29:23.516 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/linux-blkzoned.o 2023-10-11T04:29:23.677 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/cmdprio.o 2023-10-11T04:29:23.920 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/rados.o 2023-10-11T04:29:24.098 INFO:teuthology.orchestra.run.smithi040.stdout: CC engines/rbd.o 2023-10-11T04:29:24.288 INFO:teuthology.orchestra.run.smithi040.stdout: CC fio.o 2023-10-11T04:29:24.378 INFO:teuthology.orchestra.run.smithi040.stdout: LINK fio 2023-10-11T04:29:24.594 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/log.o 2023-10-11T04:29:24.637 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/genzipf.o 2023-10-11T04:29:24.782 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/strcasestr.o 2023-10-11T04:29:24.806 INFO:teuthology.orchestra.run.smithi040.stdout: CC oslib/strndup.o 2023-10-11T04:29:24.830 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/fio-genzipf 2023-10-11T04:29:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/btrace2fio.o 2023-10-11T04:29:25.360 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/fio-btrace2fio 2023-10-11T04:29:25.452 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/verify-state.o 2023-10-11T04:29:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/debug.o 2023-10-11T04:29:25.589 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/fio-verify-state 2023-10-11T04:29:25.678 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/stest.o 2023-10-11T04:29:25.748 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/arch.o 2023-10-11T04:29:25.778 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/stest 2023-10-11T04:29:25.868 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/ieee754.o 2023-10-11T04:29:25.910 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/ieee754 2023-10-11T04:29:26.000 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/axmap.o 2023-10-11T04:29:26.101 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/axmap 2023-10-11T04:29:26.189 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/lfsr-test.o 2023-10-11T04:29:26.289 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/lfsr-test 2023-10-11T04:29:26.381 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/gen-rand.o 2023-10-11T04:29:26.479 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/gen-rand 2023-10-11T04:29:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/memlock.o 2023-10-11T04:29:26.631 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/memlock 2023-10-11T04:29:26.724 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/read-to-pipe-async.o 2023-10-11T04:29:26.916 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/read-to-pipe-async 2023-10-11T04:29:27.005 INFO:teuthology.orchestra.run.smithi040.stdout: CC t/io_uring.o 2023-10-11T04:29:27.565 INFO:teuthology.orchestra.run.smithi040.stdout: LINK t/io_uring 2023-10-11T04:29:27.654 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-10-11T04:29:27.688 DEBUG:teuthology.orchestra.run.smithi040:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-11T04:29:27.933 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - Settings.cluster: 2023-10-11T04:29:27.933 INFO:teuthology.orchestra.run.smithi040.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-10-11T04:29:27.933 INFO:teuthology.orchestra.run.smithi040.stderr: 'clients': ['ubuntu@smithi040.front.sepia.ceph.com'], 2023-10-11T04:29:27.934 INFO:teuthology.orchestra.run.smithi040.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-10-11T04:29:27.934 INFO:teuthology.orchestra.run.smithi040.stderr: 'head': 'ubuntu@smithi040.front.sepia.ceph.com', 2023-10-11T04:29:27.934 INFO:teuthology.orchestra.run.smithi040.stderr: 'is_teuthology': True, 2023-10-11T04:29:27.934 INFO:teuthology.orchestra.run.smithi040.stderr: 'iterations': 1, 2023-10-11T04:29:27.935 INFO:teuthology.orchestra.run.smithi040.stderr: 'mons': ['ubuntu@smithi040.front.sepia.ceph.com'], 2023-10-11T04:29:27.935 INFO:teuthology.orchestra.run.smithi040.stderr: 'osds': ['ubuntu@smithi040.front.sepia.ceph.com'], 2023-10-11T04:29:27.935 INFO:teuthology.orchestra.run.smithi040.stderr: 'osds_per_node': 3, 2023-10-11T04:29:27.935 INFO:teuthology.orchestra.run.smithi040.stderr: 'pid_dir': '/var/run/ceph', 2023-10-11T04:29:27.935 INFO:teuthology.orchestra.run.smithi040.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2023-10-11T04:29:27.936 INFO:teuthology.orchestra.run.smithi040.stderr: 'rebuild_every_test': False, 2023-10-11T04:29:27.936 INFO:teuthology.orchestra.run.smithi040.stderr: 'rgws': {'ubuntu@smithi040.front.sepia.ceph.com': None}, 2023-10-11T04:29:27.936 INFO:teuthology.orchestra.run.smithi040.stderr: 'tmp_dir': '/tmp/cbt', 2023-10-11T04:29:27.936 INFO:teuthology.orchestra.run.smithi040.stderr: 'use_existing': True, 2023-10-11T04:29:27.936 INFO:teuthology.orchestra.run.smithi040.stderr: 'user': 'ubuntu'} 2023-10-11T04:29:27.937 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi 2023-10-11T04:29:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.938 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.939 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2023-10-11T04:29:27.940 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-10-11T04:29:27.940 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.940 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.941 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl 2023-10-11T04:29:27.942 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.943 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-11T04:29:27.946 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.946 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.947 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf 2023-10-11T04:29:27.947 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:29:27.948 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-11T04:29:27.948 INFO:teuthology.orchestra.run.smithi040.stderr:04:29:27 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-11T04:30:27.979 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:27 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:27.979 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:27 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:27.980 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-11T04:30:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.017 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-11T04:30:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.057 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* 2023-10-11T04:30:28.077 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.078 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/$(basename ${i}).smithi040.front.sepia.ceph.com ; done 2023-10-11T04:30:28.082 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.082 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.082 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl 2023-10-11T04:30:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-11T04:30:28.086 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.086 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.087 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf 2023-10-11T04:30:28.090 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.090 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-11T04:30:28.090 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-11T04:30:28.091 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.091 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.091 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2023-10-11T04:30:28.394 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.394 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:28.395 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2023-10-11T04:30:29.026 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:29 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:29.026 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:29 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:29.027 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2023-10-11T04:30:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:30 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:30 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2023-10-11T04:30:31.035 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:31 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:31.035 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:31 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2023-10-11T04:30:32.040 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:32 - INFO - cbt - Checking Health after pool creation. 2023-10-11T04:30:37.044 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:37 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:37.044 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:37 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:37.044 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-11T04:30:37.374 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:37 - INFO - cbt - Final Pool Health Check. 2023-10-11T04:30:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-11T04:30:42.726 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:42.726 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:42.727 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2023-10-11T04:30:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi040.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2023-10-11T04:30:44.409 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.409 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.409 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi040.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2023-10-11T04:30:44.506 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.506 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.507 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-11T04:30:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-11T04:30:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - INFO - cbt - Attempting to populating fio files... 2023-10-11T04:30:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.555 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.556 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-10-11T04:30:44.557 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.557 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:30:44.557 INFO:teuthology.orchestra.run.smithi040.stderr:04:30:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-10-11T04:33:17.953 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.953 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f. 2023-10-11T04:33:17.954 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - INFO - cbt - Setting OSD Read Ahead to: 4096 2023-10-11T04:33:17.954 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.954 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.954 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2023-10-11T04:33:17.955 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2023-10-11T04:33:17.955 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.955 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.956 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2023-10-11T04:33:17.974 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.974 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:17.975 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-11T04:33:18.225 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:18 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:18.225 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-11T04:33:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:18 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:18 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:18.980 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2023-10-11T04:33:19.112 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:19 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:19.112 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:19 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/ceph_settings.out 2023-10-11T04:33:45.052 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log 2023-10-11T04:33:45.419 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.419 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.419 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/collectl 2023-10-11T04:33:45.421 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.421 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-11T04:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf 2023-10-11T04:33:45.426 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.426 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-11T04:33:45.426 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-11T04:33:45.427 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - INFO - cbt - Running rbd fio randrw test. 2023-10-11T04:33:45.427 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2023-10-11T04:33:45.427 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.427 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.427 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 2023-10-11T04:33:45.428 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2023-10-11T04:33:45.428 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.428 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:33:45.429 INFO:teuthology.orchestra.run.smithi040.stderr:04:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 2023-10-11T04:34:48.005 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.005 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.006 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-11T04:34:48.087 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.088 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.088 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-11T04:34:48.128 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.128 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf.data 2023-10-11T04:34:48.130 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.130 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.* 2023-10-11T04:34:48.132 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.132 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.132 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/historic_ops.out 2023-10-11T04:34:48.497 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.775 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.776 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* 2023-10-11T04:34:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - get_fqdn_local()=smithi040.front.sepia.ceph.com 2023-10-11T04:34:48.797 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/$(basename ${i}).smithi040.front.sepia.ceph.com ; done 2023-10-11T04:34:49.035 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:49 - INFO - cbt - Convert results to json format. 2023-10-11T04:34:49.036 INFO:teuthology.orchestra.run.smithi040.stderr:04:34:49 - DEBUG - cbt - Nodes : ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:34:49.036 INFO:teuthology.orchestra.run.smithi040.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2023-10-11T04:34:49.037 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com: smithi040.front.sepia.ceph.com 2023-10-11T04:34:49.037 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-10-11T04:34:49.037 INFO:teuthology.orchestra.run.smithi040.stdout:['smithi040.front.sepia.ceph.com'] 2023-10-11T04:34:49.103 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-10-11T04:34:49.116 DEBUG:teuthology.orchestra.run.smithi040:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-10-11T04:34:49.486 DEBUG:teuthology.orchestra.run.smithi040:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2023-10-11T04:34:49.581 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-10-11T04:34:49.581 INFO:tasks.cbt_performance:reading cbt preferences 2023-10-11T04:34:49.582 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-10-11T04:34:49.669 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.1.smithi040.front.sepia.ceph.com 2023-10-11T04:34:49.669 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.0.smithi040.front.sepia.ceph.com 2023-10-11T04:34:49.670 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:34:49.671 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.1.smithi040.front.sepia.ceph.com of=/dev/stdout 2023-10-11T04:34:49.693 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-11T04:34:49.693 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f -name 'perf_stat.*' 2023-10-11T04:34:49.748 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-11T04:34:49.748 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-11T04:34:49.749 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-10-11T04:34:49.749 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.0.smithi040.front.sepia.ceph.com of=/dev/stdout 2023-10-11T04:34:49.805 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-11T04:34:49.805 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f -name 'perf_stat.*' 2023-10-11T04:34:49.860 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-11T04:34:49.861 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-11T04:34:49.871 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-11T04:34:49.880 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-11T04:34:49.880 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-11T04:34:49.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-11T04:34:50.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:34:50.028+0000 7f2709ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:34:50.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-10-11T04:34:50.037+0000 7f2709ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-11T04:34:50.260 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-11T04:34:50.261 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-11T04:34:50.261 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-11T04:34:50.262 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T04:34:55.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-11T04:34:56.363 INFO:tasks.ceph.osd.0:Stopped 2023-10-11T04:34:56.364 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-11T04:34:56.364 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T04:35:00.424 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-11T04:35:00.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-11T04:35:02.466 INFO:tasks.ceph.osd.1:Stopped 2023-10-11T04:35:02.466 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-11T04:35:02.467 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T04:35:05.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-11T04:35:05.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-11T04:35:08.568 INFO:tasks.ceph.osd.2:Stopped 2023-10-11T04:35:08.569 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-11T04:35:08.569 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-11T04:35:08.569 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T04:35:08.600 INFO:tasks.ceph.mgr.x:Stopped 2023-10-11T04:35:08.600 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-11T04:35:08.601 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-11T04:35:08.601 INFO:teuthology.orchestra.run:waiting for 300 2023-10-11T04:35:08.654 INFO:tasks.ceph.mon.a:Stopped 2023-10-11T04:35:08.655 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-11T04:35:08.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-11T04:35:08.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:35:08.757 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-11T04:35:08.970 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:35:08.971 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-11T04:35:09.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:35:09.055 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-11T04:35:09.121 INFO:tasks.ceph:Archiving mon data... 2023-10-11T04:35:09.122 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057/data/mon.a.tgz 2023-10-11T04:35:09.122 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-10-11T04:35:09.148 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Kqld5GPrlR 2023-10-11T04:35:09.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Kqld5GPrlR 2023-10-11T04:35:09.304 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Kqld5GPrlR 2023-10-11T04:35:09.415 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Kqld5GPrlR is 204KB 2023-10-11T04:35:09.467 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Kqld5GPrlR 2023-10-11T04:35:09.523 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-11T04:35:09.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-11T04:35:09.593 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-11T04:35:09.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-11T04:35:09.690 INFO:teuthology.orchestra.run.smithi040.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-11T04:35:09.694 INFO:tasks.ceph:Archiving crash dumps... 2023-10-11T04:35:09.695 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057/remote/smithi040/crash 2023-10-11T04:35:09.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-11T04:35:09.726 INFO:tasks.ceph:Compressing logs... 2023-10-11T04:35:09.727 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-11T04:35:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81606.log 2023-10-11T04:35:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-11T04:35:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82252.log 2023-10-11T04:35:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82253.log 2023-10-11T04:35:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86021.log 2023-10-11T04:35:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86069.log 2023-10-11T04:35:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.86021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85901.log 2023-10-11T04:35:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80915.log 2023-10-11T04:35:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.86069.log: gzip/var/log/ceph/ceph-client.admin.85901.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86380.log 2023-10-11T04:35:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80955.log 2023-10-11T04:35:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.80915.log: /var/log/ceph/ceph-client.admin.86380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81648.log 2023-10-11T04:35:09.816 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86380.log.gz 2023-10-11T04:35:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.80955.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.85810.log 2023-10-11T04:35:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81143.log 2023-10-11T04:35:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.85810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82339.log 2023-10-11T04:35:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.81143.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.85974.log 2023-10-11T04:35:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82381.log 2023-10-11T04:35:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.85974.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-11T04:35:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81113.log 2023-10-11T04:35:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86716.log 2023-10-11T04:35:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81113.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79318.log 2023-10-11T04:35:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.86716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82127.log 2023-10-11T04:35:09.821 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86716.log.gz 2023-10-11T04:35:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.tmp-client.admin.79318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80875.log 2023-10-11T04:35:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82087.log 2023-10-11T04:35:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.80875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81873.log 2023-10-11T04:35:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85997.log 2023-10-11T04:35:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81961.log 2023-10-11T04:35:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.85997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81793.log 2023-10-11T04:35:09.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82001.log 2023-10-11T04:35:09.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81793.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-11T04:35:09.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86428.log 2023-10-11T04:35:09.823 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82423.log 2023-10-11T04:35:09.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.86428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86427.log 2023-10-11T04:35:09.824 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86428.log.gz 2023-10-11T04:35:09.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85874.log 2023-10-11T04:35:09.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.86427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82000.log 2023-10-11T04:35:09.824 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86427.log.gz 2023-10-11T04:35:09.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.85874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81709.log 2023-10-11T04:35:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81875.log 2023-10-11T04:35:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81709.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-11T04:35:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86070.log 2023-10-11T04:35:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80677.log 2023-10-11T04:35:09.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.86070.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-11T04:35:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.80677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80765.log 2023-10-11T04:35:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81751.log 2023-10-11T04:35:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.80765.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-11T04:35:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81751.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-11T04:35:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85926.log 2023-10-11T04:35:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82213.log 2023-10-11T04:35:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.85926.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81606.log.gz 2023-10-11T04:35:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86038.log 87.6% 2023-10-11T04:35:09.827 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-10-11T04:35:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85951.log 2023-10-11T04:35:09.828 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82253.log.gz 2023-10-11T04:35:09.828 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81648.log.gz 2023-10-11T04:35:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.86038.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85974.log.gz -5 2023-10-11T04:35:09.828 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81550.log 2023-10-11T04:35:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.85951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82126.log 2023-10-11T04:35:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.81550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81874.log 2023-10-11T04:35:09.829 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80875.log.gz 2023-10-11T04:35:09.829 INFO:teuthology.orchestra.run.smithi040.stderr: 86.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-11T04:35:09.829 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85997.log.gz 2023-10-11T04:35:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.82126.log: /var/log/ceph/ceph-client.admin.81874.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81793.log.gz 2023-10-11T04:35:09.830 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81709.log.gz 2023-10-11T04:35:09.830 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81751.log.gz 2023-10-11T04:35:09.830 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81875.log.gz 2023-10-11T04:35:09.830 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81873.log.gz 2023-10-11T04:35:09.830 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81961.log.gz 2023-10-11T04:35:09.831 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80765.log.gz 2023-10-11T04:35:09.831 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81874.log.gz 2023-10-11T04:35:09.831 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82000.log.gz 2023-10-11T04:35:09.831 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82001.log.gz 2023-10-11T04:35:09.831 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82087.log.gz 2023-10-11T04:35:09.831 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82127.log.gz 2023-10-11T04:35:09.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82126.log.gz 2023-10-11T04:35:09.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82213.log.gz 2023-10-11T04:35:09.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82252.log.gz 2023-10-11T04:35:09.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82339.log.gz 2023-10-11T04:35:09.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82381.log.gz 2023-10-11T04:35:09.833 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82423.log.gz 2023-10-11T04:35:09.834 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81143.log.gz 2023-10-11T04:35:09.834 INFO:teuthology.orchestra.run.smithi040.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.80915.log.gz 2023-10-11T04:35:09.834 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80955.log.gz 2023-10-11T04:35:09.834 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81113.log.gz 2023-10-11T04:35:09.834 INFO:teuthology.orchestra.run.smithi040.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79318.log.gz 2023-10-11T04:35:09.836 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81550.log.gz 2023-10-11T04:35:09.837 INFO:teuthology.orchestra.run.smithi040.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.86021.log.gz 2023-10-11T04:35:09.838 INFO:teuthology.orchestra.run.smithi040.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.86038.log.gz 2023-10-11T04:35:09.838 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86070.log.gz 2023-10-11T04:35:09.838 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86069.log.gz 2023-10-11T04:35:09.865 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85810.log.gz 2023-10-11T04:35:09.866 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85874.log.gz 2023-10-11T04:35:09.866 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85901.log.gz 2023-10-11T04:35:09.866 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85951.log.gz 2023-10-11T04:35:09.866 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85926.log.gz 2023-10-11T04:35:09.921 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.80677.log.gz 2023-10-11T04:35:09.922 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-11T04:35:10.081 INFO:teuthology.orchestra.run.smithi040.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-11T04:35:10.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-11T04:35:28.119 INFO:teuthology.orchestra.run.smithi040.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-11T04:35:28.455 INFO:teuthology.orchestra.run.smithi040.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-11T04:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-11T04:35:28.516 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-10-11T04:35:28.516 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m18.734s 2023-10-11T04:35:28.516 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m13.778s 2023-10-11T04:35:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.656s 2023-10-11T04:35:28.517 INFO:tasks.ceph:Archiving logs... 2023-10-11T04:35:28.517 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057/remote/smithi040/log 2023-10-11T04:35:28.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-11T04:35:30.617 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-11T04:35:30.628 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-11T04:35:30.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-11T04:35:30.712 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-11T04:35:30.721 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-11T04:35:30.721 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-11T04:35:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2023-10-11T04:35:30.805 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-11T04:35:30.806 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-10-11T04:35:30.806 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 -59us[ -59us] +/- 72ms 2023-10-11T04:35:30.806 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -653us[ -660us] +/- 55ms 2023-10-11T04:35:30.806 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +886us[ +886us] +/- 75ms 2023-10-11T04:35:30.807 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-11T04:35:30.824 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-11T04:35:30.833 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-11T04:35:30.834 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-11T04:35:30.842 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-11T04:35:31.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-11T04:35:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T04:35:31.207 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-11T04:35:31.224 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-11T04:35:31.235 INFO:teuthology.task.internal:Duration was 969.510992 seconds 2023-10-11T04:35:31.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-11T04:35:31.246 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-11T04:35:31.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-11T04:35:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-11T04:35:31.575 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-11T04:35:31.576 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-10-11T04:35:31.576 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-11T04:35:31.596 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-11T04:35:31.597 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-11T04:35:31.669 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-11T04:35:31.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-11T04:35:31.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-11T04:35:31.821 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-11T04:35:31.822 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-11T04:35:31.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-11T04:35:31.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-11T04:35:31.941 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-10-11T04:35:31.968 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-11T04:35:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-11T04:35:32.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-11T04:35:32.032 INFO:teuthology.task.internal:Transferring archived files... 2023-10-11T04:35:32.033 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2023-10-11_02:17:04-crimson-rados-wip-yingxin-crimson-osd-crosscore-pg-submission-distro-default-smithi/7421057/remote/smithi040 2023-10-11T04:35:32.034 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-11T04:35:32.152 INFO:teuthology.task.internal:Removing archive directory... 2023-10-11T04:35:32.152 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-11T04:35:32.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-11T04:35:32.222 INFO:teuthology.task.internal:Not uploading archives. 2023-10-11T04:35:32.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-11T04:35:32.232 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-11T04:35:32.232 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-11T04:35:32.263 INFO:teuthology.orchestra.run.smithi040.stdout: 265423 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 11 04:35 /home/ubuntu/cephtest 2023-10-11T04:35:32.264 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-11T04:35:32.274 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-11T04:35:32.295 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-yingxin-crimson-osd-crosscore-pg-submission&var-job_id_selected=7421057 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} duration: 969.5109915733337 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2023-10-11T04:35:32.296 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-11T04:35:32.373 INFO:teuthology.run:pass