2024-04-22T06:11:49.174 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T06:11:49.174 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188 --verbose --name yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi -- /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188/orig.config.yaml 2024-04-22T06:11:49.221 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:11:49.298 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188 branch: ci-yingxin-crimson-fix-tri-mutex-4-base description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7668188' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-fix-tri-mutex-4-base ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-fix-tri-mutex-4-base env: CRIMSON_COMPAT: '1' sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-fix-tri-mutex-4-base suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxn98Jkmp/cCnJNWBuK02hv0gAdfj9lrg5UggS4GHSjy1MO13Y9kb/aUfG3A1Co3x2gWHMmKYKh3a/6R3EWvq0= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_05:57:37 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T06:11:49.298 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa; will attempt to use it 2024-04-22T06:11:49.299 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks 2024-04-22T06:11:49.299 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T06:11:49.301 INFO:teuthology.task.internal:Checking packages... 2024-04-22T06:11:49.325 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '8d82defa854b9197ed2c2b855ac5e996c480ec88' 2024-04-22T06:11:49.325 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T06:11:49.326 INFO:teuthology.packaging:ref: None 2024-04-22T06:11:49.326 INFO:teuthology.packaging:tag: None 2024-04-22T06:11:49.326 INFO:teuthology.packaging:branch: ci-yingxin-crimson-fix-tri-mutex-4-base 2024-04-22T06:11:49.326 INFO:teuthology.packaging:sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:11:49.326 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-fix-tri-mutex-4-base 2024-04-22T06:11:49.543 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3189.g8d82defa 2024-04-22T06:11:49.545 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T06:11:49.551 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T06:11:49.551 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T06:11:49.560 INFO:teuthology.task.internal:Saving configuration 2024-04-22T06:11:49.577 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T06:11:49.584 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:11:49.601 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:05:04.655801', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxn98Jkmp/cCnJNWBuK02hv0gAdfj9lrg5UggS4GHSjy1MO13Y9kb/aUfG3A1Co3x2gWHMmKYKh3a/6R3EWvq0='} 2024-04-22T06:11:49.601 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T06:11:49.609 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-22T06:11:49.609 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T06:11:49.643 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1309fc6e50>, signals=[15]) 2024-04-22T06:11:49.643 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T06:11:49.650 INFO:teuthology.task.internal:Opening connections... 2024-04-22T06:11:49.650 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:11:49.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:11:49.731 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T06:11:49.740 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-04-22T06:11:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-04-22T06:11:49.817 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:11:49.876 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:11:49.877 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:11:49.877 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:11:49.877 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:11:49.877 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:11:49.877 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-04-22T06:11:49.949 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T06:11:49.958 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T06:11:49.966 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T06:11:49.966 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:11:49.985 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T06:11:49.994 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T06:11:49.994 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:11:50.053 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T06:11:50.088 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T06:11:50.104 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-22T06:11:50.104 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-22T06:11:50.104 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:11:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:11:50.121 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-04-22T06:11:50.179 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:11:50.179 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-437.el9.x86_64 2024-04-22T06:11:50.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-04-22T06:12:00.522 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 945 kB/s | 8.0 MB 00:08 2024-04-22T06:12:02.390 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T06:12:06.990 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-22T06:12:08.618 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-22T06:12:10.569 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-22T06:12:15.811 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-22T06:12:17.700 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:12:17.701 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:12:17.784 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:12:17.785 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:12:17.785 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:12:17.909 DEBUG:teuthology.orchestra.run.smithi049:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:12:18.430 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:12:15 AM UTC. 2024-04-22T06:12:18.562 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:12:18.566 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:12:18.566 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout:Reinstalling: 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:12:18.567 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:12:18.568 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 255 k 2024-04-22T06:12:18.568 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 0 2024-04-22T06:12:18.568 INFO:teuthology.orchestra.run.smithi049.stderr:Operation aborted. 2024-04-22T06:12:18.636 INFO:teuthology.orchestra.run.smithi049.stdout:Is this ok [y/N]: 2024-04-22T06:12:18.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum reinstall -y kernel || true 2024-04-22T06:12:19.154 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:04 ago on Mon 22 Apr 2024 06:12:15 AM UTC. 2024-04-22T06:12:19.274 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:Reinstalling: 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 255 k 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 0 2024-04-22T06:12:19.275 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:12:19.655 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.0 MB/s | 255 kB 00:00 2024-04-22T06:12:19.655 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:12:19.655 INFO:teuthology.orchestra.run.smithi049.stdout:Total 671 kB/s | 255 kB 00:00 2024-04-22T06:12:19.674 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:12:19.850 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:12:19.850 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:12:19.855 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:12:19.856 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:12:20.002 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:12:20.072 INFO:teuthology.orchestra.run.smithi049.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:12:20.073 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:12:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:12:20.776 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:12:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:12:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:12:21.473 INFO:teuthology.orchestra.run.smithi049.stdout:Reinstalled: 2024-04-22T06:12:21.473 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:12:21.473 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:12:21.473 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:12:21.653 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:12:21.741 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:12:21.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:12:21.742 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:12:21.742 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T06:12:21.742 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T06:12:21.772 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-04-22T06:12:21.797 DEBUG:teuthology.parallel:result is None 2024-04-22T06:12:21.798 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T06:12:21.809 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T06:12:21.810 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:12:21.851 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T06:12:21.862 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T06:12:21.876 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T06:12:21.876 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:12:21.921 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T06:12:21.932 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T06:12:21.932 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:12:21.988 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:12:21.997 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:12:21.999 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T06:12:22.008 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T06:12:22.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:12:22.064 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T06:12:22.074 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T06:12:22.074 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:12:22.120 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:12:22.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:12:22.256 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:12:22.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:12:22.378 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:12:22.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:12:22.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-04-22T06:12:22.527 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:12:22.948 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T06:12:23.002 INFO:teuthology.task.internal:Starting timer... 2024-04-22T06:12:23.002 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T06:12:23.015 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T06:12:23.028 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-22T06:12:23.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-04-22T06:12:23.080 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-04-22T06:12:23.082 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T06:12:23.082 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-04-22T06:12:23.109 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-04-22T06:12:23.109 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive'} 2024-04-22T06:12:23.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:12:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:12:23.175 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T06:12:23.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:12:23.240 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T06:12:23.251 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T06:12:23.270 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T06:12:23.272 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T06:15:58.098 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2024-04-22T06:15:58.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T06:15:58.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:15:58.224 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-04-22T06:15:58.301 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-22T06:15:58.302 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T06:15:58.314 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T06:15:58.314 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:15:58.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:15:58.371 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:15:58.384 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:15:58.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-04-22T06:15:58.419 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-04-22T06:15:58.433 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:15:58.446 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:15:58.506 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:15:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:15:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-04-22T06:15:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:15:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:15:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:15:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:15:58.512 INFO:teuthology.run_tasks:Running task install... 2024-04-22T06:15:58.520 DEBUG:teuthology.task.install:project ceph 2024-04-22T06:15:58.520 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}} 2024-04-22T06:15:58.520 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'} 2024-04-22T06:15:58.520 INFO:teuthology.task.install:Using flavor: crimson 2024-04-22T06:15:58.526 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T06:15:58.526 INFO:teuthology.task.install:extra packages: [] 2024-04-22T06:15:58.527 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'tag': None, 'wait_for_package': False} 2024-04-22T06:15:58.527 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:15:58.756 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/ 2024-04-22T06:15:58.756 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3189.g8d82defa 2024-04-22T06:15:58.936 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:15:58.936 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:15:58.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:15:58.962 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T06:15:58.962 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8d82defa854b9197ed2c2b855ac5e996c480ec88/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:15:59.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:15:59.105 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:15:59.168 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-04-22T06:15:59.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-04-22T06:15:59.404 INFO:teuthology.orchestra.run.smithi049.stdout:48 files removed 2024-04-22T06:15:59.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2024-04-22T06:16:00.163 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 189 kB/s | 87 kB 00:00 2024-04-22T06:16:00.546 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-22T06:16:01.148 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 3.6 kB/s | 2.0 kB 00:00 2024-04-22T06:16:01.781 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-22T06:16:03.641 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-22T06:16:08.008 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.2 MB 00:00 2024-04-22T06:16:09.835 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-04-22T06:16:23.401 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 1.6 MB/s | 21 MB 00:13 2024-04-22T06:16:28.617 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-22T06:16:30.866 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:16:30.868 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:16:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:16:30.868 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:16:30.868 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:16:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 10 M 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.7 M 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.9 M 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 19 M 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 25 k 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 796 k 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:16:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 547 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.3 M 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 45 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 135 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 159 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 315 k 2024-04-22T06:16:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 293 k 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 99 k 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout:Install 30 Packages 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2024-04-22T06:16:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:30.872 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 59 M 2024-04-22T06:16:30.872 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:16:32.260 INFO:teuthology.orchestra.run.smithi049.stdout:(1/32): ceph-radosgw-19.0.0-3189.g8d82defa.el9. 9.8 MB/s | 10 MB 00:01 2024-04-22T06:16:32.318 INFO:teuthology.orchestra.run.smithi049.stdout:(2/32): ceph-selinux-19.0.0-3189.g8d82defa.el9. 433 kB/s | 25 kB 00:00 2024-04-22T06:16:32.435 INFO:teuthology.orchestra.run.smithi049.stdout:(3/32): ceph-base-19.0.0-3189.g8d82defa.el9.x86 4.9 MB/s | 5.9 MB 00:01 2024-04-22T06:16:32.469 INFO:teuthology.orchestra.run.smithi049.stdout:(4/32): libcephfs2-19.0.0-3189.g8d82defa.el9.x8 5.2 MB/s | 796 kB 00:00 2024-04-22T06:16:32.560 INFO:teuthology.orchestra.run.smithi049.stdout:(5/32): libradosstriper1-19.0.0-3189.g8d82defa. 4.3 MB/s | 547 kB 00:00 2024-04-22T06:16:32.644 INFO:teuthology.orchestra.run.smithi049.stdout:(6/32): python3-ceph-argparse-19.0.0-3189.g8d82 540 kB/s | 45 kB 00:00 2024-04-22T06:16:32.727 INFO:teuthology.orchestra.run.smithi049.stdout:(7/32): python3-ceph-common-19.0.0-3189.g8d82de 1.6 MB/s | 135 kB 00:00 2024-04-22T06:16:32.836 INFO:teuthology.orchestra.run.smithi049.stdout:(8/32): librgw2-19.0.0-3189.g8d82defa.el9.x86_6 14 MB/s | 5.3 MB 00:00 2024-04-22T06:16:32.861 INFO:teuthology.orchestra.run.smithi049.stdout:(9/32): python3-cephfs-19.0.0-3189.g8d82defa.el 1.2 MB/s | 159 kB 00:00 2024-04-22T06:16:32.895 INFO:teuthology.orchestra.run.smithi049.stdout:(10/32): python3-rados-19.0.0-3189.g8d82defa.el 5.3 MB/s | 315 kB 00:00 2024-04-22T06:16:32.945 INFO:teuthology.orchestra.run.smithi049.stdout:(11/32): python3-rbd-19.0.0-3189.g8d82defa.el9. 3.4 MB/s | 293 kB 00:00 2024-04-22T06:16:32.971 INFO:teuthology.orchestra.run.smithi049.stdout:(12/32): python3-rgw-19.0.0-3189.g8d82defa.el9. 1.3 MB/s | 99 kB 00:00 2024-04-22T06:16:33.062 INFO:teuthology.orchestra.run.smithi049.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 914 kB/s | 107 kB 00:00 2024-04-22T06:16:33.088 INFO:teuthology.orchestra.run.smithi049.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-22T06:16:33.113 INFO:teuthology.orchestra.run.smithi049.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-04-22T06:16:33.189 INFO:teuthology.orchestra.run.smithi049.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-04-22T06:16:33.214 INFO:teuthology.orchestra.run.smithi049.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-04-22T06:16:33.239 INFO:teuthology.orchestra.run.smithi049.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 837 kB/s | 42 kB 00:00 2024-04-22T06:16:33.264 INFO:teuthology.orchestra.run.smithi049.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-22T06:16:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 715 kB/s | 36 kB 00:00 2024-04-22T06:16:33.315 INFO:teuthology.orchestra.run.smithi049.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-22T06:16:33.516 INFO:teuthology.orchestra.run.smithi049.stdout:(22/32): ceph-common-19.0.0-3189.g8d82defa.el9. 8.5 MB/s | 19 MB 00:02 2024-04-22T06:16:33.641 INFO:teuthology.orchestra.run.smithi049.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-04-22T06:16:33.741 INFO:teuthology.orchestra.run.smithi049.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 685 kB/s | 309 kB 00:00 2024-04-22T06:16:33.820 INFO:teuthology.orchestra.run.smithi049.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 271 kB/s | 49 kB 00:00 2024-04-22T06:16:33.858 INFO:teuthology.orchestra.run.smithi049.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-04-22T06:16:33.917 INFO:teuthology.orchestra.run.smithi049.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-22T06:16:34.009 INFO:teuthology.orchestra.run.smithi049.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-22T06:16:34.076 INFO:teuthology.orchestra.run.smithi049.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-04-22T06:16:34.143 INFO:teuthology.orchestra.run.smithi049.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-04-22T06:16:34.427 INFO:teuthology.orchestra.run.smithi049.stdout:(31/32): librbd1-19.0.0-3189.g8d82defa.el9.x86_ 11 MB/s | 3.7 MB 00:00 2024-04-22T06:16:34.552 INFO:teuthology.orchestra.run.smithi049.stdout:(32/32): librados2-19.0.0-3189.g8d82defa.el9.x8 6.3 MB/s | 3.4 MB 00:00 2024-04-22T06:16:34.554 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:34.554 INFO:teuthology.orchestra.run.smithi049.stdout:Total 16 MB/s | 59 MB 00:03 2024-04-22T06:16:34.651 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:16:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:16:34.691 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:16:35.178 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:16:35.540 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:16:36.703 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:16:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T06:16:37.279 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:16:37.372 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:16:37.463 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:16:37.498 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:16:37.544 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:16:37.582 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T06:16:37.610 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T06:16:37.636 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T06:16:37.658 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:16:37.680 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:16:37.877 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:16:37.909 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:16:38.285 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T06:16:38.337 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:16:38.404 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:16:38.447 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 12/34 2024-04-22T06:16:38.504 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:16:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 14/34 2024-04-22T06:16:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T06:16:38.640 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:16:38.720 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:16:38.756 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T06:16:38.868 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T06:16:38.898 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T06:16:38.938 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T06:16:38.964 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T06:16:38.993 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T06:16:39.024 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T06:16:39.061 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T06:16:39.657 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T06:16:39.697 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:16:39.768 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:16:39.840 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 27/34 2024-04-22T06:16:39.875 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 28/34 2024-04-22T06:16:42.442 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:16:42.474 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:16:43.173 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:16:43.209 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:16:43.284 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:16:43.284 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:16:43.284 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:16:43.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:43.318 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2024-04-22T06:16:53.623 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-04-22T06:16:53.624 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:54.728 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:16:54.753 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:16:54.753 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:16:54.753 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:16:54.753 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:16:54.753 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:16:54.753 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:54.755 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:16:54.831 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:16:54.831 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:16:56.229 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 1/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 5/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 6/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 7/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 8/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 9/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 10/34 2024-04-22T06:16:56.230 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 12/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T06:16:56.231 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T06:16:56.232 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 33/34 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.977 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:16:56.978 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:56.979 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:16:57.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2024-04-22T06:16:57.785 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:29 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:16:57.910 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:16:57.911 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:16:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:16:57.911 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:16:57.911 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:16:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 84 M 2024-04-22T06:16:57.911 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 173 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout:Install 9 Packages 2024-04-22T06:16:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:16:57.913 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 87 M 2024-04-22T06:16:57.913 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 813 M 2024-04-22T06:16:57.913 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:16:58.485 INFO:teuthology.orchestra.run.smithi049.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 421 kB/s | 94 kB 00:00 2024-04-22T06:16:58.543 INFO:teuthology.orchestra.run.smithi049.stdout:(2/9): libcephsqlite-19.0.0-3189.g8d82defa.el9. 615 kB/s | 173 kB 00:00 2024-04-22T06:16:58.661 INFO:teuthology.orchestra.run.smithi049.stdout:(3/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.6 MB/s | 469 kB 00:00 2024-04-22T06:16:58.692 INFO:teuthology.orchestra.run.smithi049.stdout:(4/9): libubsan-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-22T06:16:58.762 INFO:teuthology.orchestra.run.smithi049.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-22T06:16:58.788 INFO:teuthology.orchestra.run.smithi049.stdout:(6/9): socat-1.7.4.1-5.el9.x86_64.rpm 3.1 MB/s | 305 kB 00:00 2024-04-22T06:16:58.830 INFO:teuthology.orchestra.run.smithi049.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 949 kB/s | 64 kB 00:00 2024-04-22T06:16:59.217 INFO:teuthology.orchestra.run.smithi049.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 284 kB/s | 122 kB 00:00 2024-04-22T06:17:02.929 INFO:teuthology.orchestra.run.smithi049.stdout:(9/9): ceph-test-19.0.0-3189.g8d82defa.el9.x86_ 18 MB/s | 84 MB 00:04 2024-04-22T06:17:02.930 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:17:02.931 INFO:teuthology.orchestra.run.smithi049.stdout:Total 17 MB/s | 87 MB 00:05 2024-04-22T06:17:03.014 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:17:03.040 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:17:03.040 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:17:03.668 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:17:03.668 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:17:04.581 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:17:04.661 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T06:17:04.706 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T06:17:04.751 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T06:17:04.776 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T06:17:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:17:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:17:04.910 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:17:04.958 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T06:17:04.984 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:17:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:17:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:17:23.146 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:17:23.146 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 1/9 2024-04-22T06:17:23.146 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2/9 2024-04-22T06:17:23.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T06:17:23.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T06:17:23.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:17:23.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T06:17:23.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T06:17:23.147 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:17:23.532 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:23.533 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:17:23.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2024-04-22T06:17:24.356 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:56 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:17:24.484 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:17:24.486 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 6.5 k 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 2.2 M 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 1.6 M 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 249 k 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.0 M 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 71 M 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:17:24.487 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:17:24.488 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:24.489 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:17:24.490 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:17:24.490 INFO:teuthology.orchestra.run.smithi049.stdout:Install 36 Packages 2024-04-22T06:17:24.490 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:24.490 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 84 M 2024-04-22T06:17:24.490 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 923 M 2024-04-22T06:17:24.490 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:17:25.377 INFO:teuthology.orchestra.run.smithi049.stdout:(1/36): ceph-19.0.0-3189.g8d82defa.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2024-04-22T06:17:26.901 INFO:teuthology.orchestra.run.smithi049.stdout:(2/36): ceph-mgr-19.0.0-3189.g8d82defa.el9.x86_ 876 kB/s | 1.6 MB 00:01 2024-04-22T06:17:27.202 INFO:teuthology.orchestra.run.smithi049.stdout:(3/36): ceph-mds-19.0.0-3189.g8d82defa.el9.x86_ 1.0 MB/s | 2.2 MB 00:02 2024-04-22T06:17:27.402 INFO:teuthology.orchestra.run.smithi049.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3189.g8d82 1.2 MB/s | 249 kB 00:00 2024-04-22T06:17:27.577 INFO:teuthology.orchestra.run.smithi049.stdout:(5/36): ceph-mon-19.0.0-3189.g8d82defa.el9.x86_ 2.3 MB/s | 5.0 MB 00:02 2024-04-22T06:17:27.753 INFO:teuthology.orchestra.run.smithi049.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 723 kB/s | 253 kB 00:00 2024-04-22T06:17:27.853 INFO:teuthology.orchestra.run.smithi049.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-22T06:17:27.961 INFO:teuthology.orchestra.run.smithi049.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-04-22T06:17:28.070 INFO:teuthology.orchestra.run.smithi049.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-04-22T06:17:28.145 INFO:teuthology.orchestra.run.smithi049.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 2.2 MB/s | 1.2 MB 00:00 2024-04-22T06:17:28.204 INFO:teuthology.orchestra.run.smithi049.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.6 MB/s | 215 kB 00:00 2024-04-22T06:17:28.363 INFO:teuthology.orchestra.run.smithi049.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 869 kB/s | 188 kB 00:00 2024-04-22T06:17:28.388 INFO:teuthology.orchestra.run.smithi049.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 937 kB/s | 172 kB 00:00 2024-04-22T06:17:28.422 INFO:teuthology.orchestra.run.smithi049.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 593 kB/s | 35 kB 00:00 2024-04-22T06:17:28.447 INFO:teuthology.orchestra.run.smithi049.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-04-22T06:17:28.481 INFO:teuthology.orchestra.run.smithi049.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-04-22T06:17:28.550 INFO:teuthology.orchestra.run.smithi049.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 216 kB/s | 22 kB 00:00 2024-04-22T06:17:28.673 INFO:teuthology.orchestra.run.smithi049.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 788 kB/s | 151 kB 00:00 2024-04-22T06:17:28.698 INFO:teuthology.orchestra.run.smithi049.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 295 kB/s | 43 kB 00:00 2024-04-22T06:17:28.790 INFO:teuthology.orchestra.run.smithi049.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-22T06:17:28.865 INFO:teuthology.orchestra.run.smithi049.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-04-22T06:17:28.891 INFO:teuthology.orchestra.run.smithi049.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-22T06:17:28.933 INFO:teuthology.orchestra.run.smithi049.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2024-04-22T06:17:28.966 INFO:teuthology.orchestra.run.smithi049.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-04-22T06:17:29.000 INFO:teuthology.orchestra.run.smithi049.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-04-22T06:17:29.042 INFO:teuthology.orchestra.run.smithi049.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-04-22T06:17:29.142 INFO:teuthology.orchestra.run.smithi049.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 326 kB/s | 46 kB 00:00 2024-04-22T06:17:29.167 INFO:teuthology.orchestra.run.smithi049.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-04-22T06:17:29.251 INFO:teuthology.orchestra.run.smithi049.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-22T06:17:29.301 INFO:teuthology.orchestra.run.smithi049.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-04-22T06:17:29.327 INFO:teuthology.orchestra.run.smithi049.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-22T06:17:29.369 INFO:teuthology.orchestra.run.smithi049.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-04-22T06:17:29.444 INFO:teuthology.orchestra.run.smithi049.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-22T06:17:29.503 INFO:teuthology.orchestra.run.smithi049.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 344 kB/s | 20 kB 00:00 2024-04-22T06:17:29.536 INFO:teuthology.orchestra.run.smithi049.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-04-22T06:17:41.002 INFO:teuthology.orchestra.run.smithi049.stdout:(36/36): ceph-osd-19.0.0-3189.g8d82defa.el9.x86 5.0 MB/s | 71 MB 00:14 2024-04-22T06:17:41.003 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:17:41.004 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.1 MB/s | 84 MB 00:16 2024-04-22T06:17:41.146 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:17:41.191 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:17:41.192 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:17:41.836 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:17:41.837 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:17:43.451 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:17:43.512 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T06:17:43.554 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T06:17:43.581 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T06:17:43.655 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T06:17:43.698 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T06:17:43.730 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T06:17:43.827 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T06:17:43.904 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T06:17:43.975 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T06:17:44.010 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T06:17:44.040 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T06:17:44.074 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T06:17:44.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T06:17:44.149 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T06:17:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T06:17:44.226 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T06:17:44.264 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T06:17:44.342 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T06:17:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T06:17:44.576 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T06:17:44.628 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T06:17:44.665 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T06:17:44.723 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T06:17:44.787 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T06:17:44.985 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T06:17:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T06:17:45.116 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T06:17:45.205 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T06:17:45.243 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T06:17:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T06:17:45.560 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 31/36 2024-04-22T06:17:45.586 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:17:45.612 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:17:45.613 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:17:45.613 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:17:45.613 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:17:45.613 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:17:45.613 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:57.077 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:17:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:17:57.105 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:17:57.105 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:17:57.105 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:17:57.105 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:17:57.105 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:17:57.696 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:17:57.696 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:17:57.696 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:17:57.696 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:17:57.696 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:17:57.696 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:17:57.972 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:17:57.972 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:17:57.972 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:17:57.972 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:17:57.972 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:17:57.972 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:17:58.035 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:18:04.769 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:18:04.769 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 1/36 2024-04-22T06:18:04.769 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 3/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 4/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 5/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 6/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T06:18:04.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T06:18:04.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T06:18:04.772 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T06:18:05.709 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T06:18:05.709 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:05.709 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:05.709 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:18:05.710 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:18:05.711 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:18:05.712 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:18:05.713 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:18:05.713 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:18:05.713 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:18:05.713 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:18:05.713 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:05.713 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:06.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2024-04-22T06:18:06.593 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:06.674 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:06.719 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:06.720 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:06.720 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:06.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2024-04-22T06:18:07.309 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:39 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:07.434 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 762 k 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:07.435 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:07.436 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:18:07.436 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:07.436 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 762 k 2024-04-22T06:18:07.436 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 769 k 2024-04-22T06:18:07.436 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:08.212 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-19.0.0-3189.g8d82defa.el9.noarch.rpm 982 kB/s | 762 kB 00:00 2024-04-22T06:18:08.212 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:08.213 INFO:teuthology.orchestra.run.smithi049.stdout:Total 980 kB/s | 762 kB 00:00 2024-04-22T06:18:08.213 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:08.219 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:08.219 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:08.225 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:08.226 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:08.301 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:08.727 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:18:08.809 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:18:09.144 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:18:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:18:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:09.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T06:18:10.119 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:42 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:10.246 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 157 k 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:10.247 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:10.248 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:10.248 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:18:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:10.248 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 157 k 2024-04-22T06:18:10.248 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 640 k 2024-04-22T06:18:10.248 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:10.611 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-19.0.0-3189.g8d82de 433 kB/s | 157 kB 00:00 2024-04-22T06:18:10.611 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:10.612 INFO:teuthology.orchestra.run.smithi049.stdout:Total 431 kB/s | 157 kB 00:00 2024-04-22T06:18:10.612 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:10.622 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:10.623 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:10.682 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:10.683 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:10.843 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:10.920 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:18:10.946 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:18:10.946 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:18:10.946 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:18:10.946 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:11.846 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:18:11.847 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:11.847 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:11.847 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:11.847 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:11.847 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:12.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2024-04-22T06:18:12.587 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:44 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:12.667 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:12.712 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:12.713 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:12.713 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:12.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T06:18:13.294 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:45 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:13.417 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:13.418 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:13.418 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:13.418 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:13.418 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:13.418 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 3.5 M 2024-04-22T06:18:13.418 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 34 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 16 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:13.419 INFO:teuthology.orchestra.run.smithi049.stdout:Install 11 Packages 2024-04-22T06:18:13.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:13.420 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 7.6 M 2024-04-22T06:18:13.420 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 100 M 2024-04-22T06:18:13.420 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:14.007 INFO:teuthology.orchestra.run.smithi049.stdout:(1/11): ceph-grafana-dashboards-19.0.0-3189.g8d 174 kB/s | 34 kB 00:00 2024-04-22T06:18:14.032 INFO:teuthology.orchestra.run.smithi049.stdout:(2/11): ceph-prometheus-alerts-19.0.0-3189.g8d8 74 kB/s | 16 kB 00:00 2024-04-22T06:18:14.232 INFO:teuthology.orchestra.run.smithi049.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.2 MB/s | 863 kB 00:00 2024-04-22T06:18:14.316 INFO:teuthology.orchestra.run.smithi049.stdout:(4/11): ceph-mgr-dashboard-19.0.0-3189.g8d82def 7.0 MB/s | 3.5 MB 00:00 2024-04-22T06:18:14.508 INFO:teuthology.orchestra.run.smithi049.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-04-22T06:18:14.600 INFO:teuthology.orchestra.run.smithi049.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 68 kB/s | 19 kB 00:00 2024-04-22T06:18:14.625 INFO:teuthology.orchestra.run.smithi049.stdout:(7/11): python3-protobuf-3.14.0-13.el9.noarch.r 436 kB/s | 269 kB 00:00 2024-04-22T06:18:14.742 INFO:teuthology.orchestra.run.smithi049.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-04-22T06:18:14.784 INFO:teuthology.orchestra.run.smithi049.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.4 MB/s | 2.0 MB 00:00 2024-04-22T06:18:14.809 INFO:teuthology.orchestra.run.smithi049.stdout:(10/11): python3-repoze-lru-0.7-16.el9.noarch.r 167 kB/s | 31 kB 00:00 2024-04-22T06:18:14.851 INFO:teuthology.orchestra.run.smithi049.stdout:(11/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-04-22T06:18:14.853 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:14.853 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.3 MB/s | 7.6 MB 00:01 2024-04-22T06:18:14.883 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:14.902 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:14.902 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:15.059 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:15.059 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:15.547 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:15.642 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T06:18:15.671 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T06:18:15.752 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T06:18:15.887 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T06:18:15.942 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T06:18:16.030 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T06:18:16.086 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T06:18:16.139 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:18:16.172 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 9/11 2024-04-22T06:18:17.283 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 10/11 2024-04-22T06:18:17.321 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 1/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 2/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 3/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T06:18:18.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T06:18:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T06:18:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:18:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T06:18:18.604 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T06:18:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T06:18:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:18.901 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:18:18.901 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:18.902 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:19.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T06:18:19.669 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:51 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:19.795 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 7.4 M 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:18:19.797 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2024-04-22T06:18:19.798 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:19.799 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 38 M 2024-04-22T06:18:19.799 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 209 M 2024-04-22T06:18:19.799 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:20.743 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): ceph-mgr-diskprediction-local-19.0.0-31 10 MB/s | 7.4 MB 00:00 2024-04-22T06:18:20.909 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 196 kB/s | 33 kB 00:00 2024-04-22T06:18:20.935 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 213 kB/s | 195 kB 00:00 2024-04-22T06:18:21.076 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 129 kB/s | 18 kB 00:00 2024-04-22T06:18:21.128 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-04-22T06:18:21.169 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 383 kB/s | 35 kB 00:00 2024-04-22T06:18:21.236 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-22T06:18:21.412 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-22T06:18:21.436 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 565 kB/s | 803 kB 00:01 2024-04-22T06:18:21.512 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 12 MB/s | 4.7 MB 00:00 2024-04-22T06:18:21.537 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-04-22T06:18:21.989 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-22T06:18:21.990 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:21.990 INFO:teuthology.orchestra.run.smithi049.stdout:Total 17 MB/s | 38 MB 00:02 2024-04-22T06:18:22.350 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:22.377 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:22.377 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:22.812 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:22.812 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:23.651 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:23.717 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T06:18:23.777 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:18:23.889 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:18:23.930 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T06:18:24.114 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T06:18:24.144 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T06:18:24.214 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T06:18:24.910 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T06:18:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T06:18:26.337 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:18:27.414 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T06:18:27.444 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:18:29.910 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 1/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T06:18:29.911 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T06:18:29.912 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T06:18:29.912 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T06:18:29.912 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:18:29.912 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T06:18:30.337 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T06:18:30.337 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:30.337 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:30.337 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:30.337 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:30.338 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:30.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2024-04-22T06:18:31.231 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:03 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:31.358 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 49 k 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:18:31.359 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:31.360 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:31.361 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2024-04-22T06:18:31.361 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:31.361 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.2 M 2024-04-22T06:18:31.361 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2024-04-22T06:18:31.361 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:31.720 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 19 kB 00:00 2024-04-22T06:18:31.745 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 26 kB 00:00 2024-04-22T06:18:31.774 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): ceph-mgr-rook-19.0.0-3189.g8d82defa.el9 259 kB/s | 49 kB 00:00 2024-04-22T06:18:31.879 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-22T06:18:31.905 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-04-22T06:18:31.930 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-04-22T06:18:31.989 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-04-22T06:18:32.156 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2024-04-22T06:18:32.181 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-04-22T06:18:32.373 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 528 kB/s | 203 kB 00:00 2024-04-22T06:18:32.415 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-04-22T06:18:32.532 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-04-22T06:18:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-04-22T06:18:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:32.641 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-04-22T06:18:32.713 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:32.730 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:32.823 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:32.823 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:33.348 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T06:18:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T06:18:33.733 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T06:18:33.768 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T06:18:33.810 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T06:18:33.915 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T06:18:34.026 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T06:18:34.079 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T06:18:34.841 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T06:18:34.892 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T06:18:34.925 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T06:18:34.975 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T06:18:35.003 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 1/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T06:18:35.641 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T06:18:35.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T06:18:35.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T06:18:35.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T06:18:35.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T06:18:35.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T06:18:35.642 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:18:35.967 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:35.968 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:36.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T06:18:36.756 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:08 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:36.882 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:36.882 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:36.882 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 142 k 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2024-04-22T06:18:36.883 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:36.884 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 7.0 M 2024-04-22T06:18:36.884 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 33 M 2024-04-22T06:18:36.884 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:37.501 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3189.g8d82defa.e 298 kB/s | 142 kB 00:00 2024-04-22T06:18:37.610 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 423 kB/s | 248 kB 00:00 2024-04-22T06:18:37.693 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2024-04-22T06:18:37.719 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 531 kB/s | 58 kB 00:00 2024-04-22T06:18:37.736 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-04-22T06:18:38.965 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.1 MB/s | 6.0 MB 00:01 2024-04-22T06:18:38.965 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:38.965 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.4 MB/s | 7.0 MB 00:02 2024-04-22T06:18:39.042 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:39.054 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:39.054 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:39.158 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:39.159 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:39.760 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T06:18:39.806 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T06:18:40.370 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T06:18:40.461 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T06:18:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T06:18:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:18:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:18:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/6 2024-04-22T06:18:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T06:18:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T06:18:41.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T06:18:41.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T06:18:41.548 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:41.549 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:41.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2024-04-22T06:18:42.284 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:14 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:42.408 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:42.409 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:42.409 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:42.409 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 871 k 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-04-22T06:18:42.410 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 951 k 2024-04-22T06:18:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 4.0 M 2024-04-22T06:18:42.411 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:42.973 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-fuse-19.0.0-3189.g8d82defa.el9.x86_ 1.9 MB/s | 871 kB 00:00 2024-04-22T06:18:43.215 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 118 kB/s | 80 kB 00:00 2024-04-22T06:18:43.216 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:43.216 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.2 MB/s | 951 kB 00:00 2024-04-22T06:18:43.227 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:43.252 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:43.253 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:43.335 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:43.335 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:43.524 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:43.680 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T06:18:43.765 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:18:44.453 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/2 2024-04-22T06:18:44.722 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T06:18:44.723 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:44.723 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:44.723 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T06:18:44.723 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:44.723 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:44.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2024-04-22T06:18:45.508 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:17 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:45.633 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 273 k 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:18:45.634 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 350 k 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.5 M 2024-04-22T06:18:45.635 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:46.103 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 398 kB/s | 77 kB 00:00 2024-04-22T06:18:46.153 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-volume-19.0.0-3189.g8d82defa.el9.no 1.1 MB/s | 273 kB 00:00 2024-04-22T06:18:46.154 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:46.154 INFO:teuthology.orchestra.run.smithi049.stdout:Total 675 kB/s | 350 kB 00:00 2024-04-22T06:18:46.166 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:46.176 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:46.176 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:46.205 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:46.205 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T06:18:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:18:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:18:46.597 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:18:46.597 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:18:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:47.270 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 1/2 2024-04-22T06:18:47.558 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T06:18:47.558 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:47.558 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:47.558 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:18:47.558 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:18:47.559 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:47.559 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:47.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2024-04-22T06:18:48.396 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:20 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:48.521 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 125 k 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:18:48.522 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:48.523 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 125 k 2024-04-22T06:18:48.523 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 445 k 2024-04-22T06:18:48.523 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:49.008 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-19.0.0-3189.g8d82defa.el9.x86_64 257 kB/s | 125 kB 00:00 2024-04-22T06:18:49.009 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:49.009 INFO:teuthology.orchestra.run.smithi049.stdout:Total 257 kB/s | 125 kB 00:00 2024-04-22T06:18:49.010 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:49.016 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:49.016 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:49.085 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:49.086 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:49.241 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:49.350 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:49.793 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:50.047 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:50.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:50.047 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:50.047 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:50.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:50.048 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:50.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2024-04-22T06:18:50.809 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:50.894 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:50.935 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:50.936 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:50.936 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:51.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2024-04-22T06:18:51.543 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:23 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:51.668 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:51.668 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 32 k 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 32 k 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 142 k 2024-04-22T06:18:51.669 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:51.954 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-19.0.0-3189.g8d82defa.el9.x86_6 112 kB/s | 32 kB 00:00 2024-04-22T06:18:51.954 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:51.955 INFO:teuthology.orchestra.run.smithi049.stdout:Total 111 kB/s | 32 kB 00:00 2024-04-22T06:18:51.955 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:51.958 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:51.958 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:51.961 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:51.961 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:52.050 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:52.161 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:52.513 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:52.737 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:52.737 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:52.737 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:52.737 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:52.737 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:52.738 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:52.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2024-04-22T06:18:53.425 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:53.507 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:53.551 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:53.552 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:53.552 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:53.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2024-04-22T06:18:54.145 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:26 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:54.226 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:54.270 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:54.271 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:54.271 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:54.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2024-04-22T06:18:54.869 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:26 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:54.950 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:54.994 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:54.995 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:54.995 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:55.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2024-04-22T06:18:55.599 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:27 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:55.681 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:55.725 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:55.726 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:55.726 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:55.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2024-04-22T06:18:56.322 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:28 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:56.405 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:56.449 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:56.450 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:56.450 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:56.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2024-04-22T06:18:57.044 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:29 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:57.126 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:18:57.168 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:57.170 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-04-22T06:18:57.170 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:57.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2024-04-22T06:18:57.775 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:29 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:57.901 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 90 k 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:18:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:57.903 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 90 k 2024-04-22T06:18:57.903 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 313 k 2024-04-22T06:18:57.903 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:18:58.107 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-19.0.0-3189.g8d82defa.el9.x86_64.rpm 440 kB/s | 90 kB 00:00 2024-04-22T06:18:58.108 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:18:58.108 INFO:teuthology.orchestra.run.smithi049.stdout:Total 436 kB/s | 90 kB 00:00 2024-04-22T06:18:58.109 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:18:58.118 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:18:58.119 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:18:58.186 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:18:58.186 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:18:58.336 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:18:58.448 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:58.861 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:59.113 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:18:59.113 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:59.113 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:18:59.113 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:18:59.113 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:59.113 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:18:59.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2024-04-22T06:18:59.867 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:31 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:18:59.991 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:18:59.992 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:18:59.993 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.4 M 2024-04-22T06:18:59.993 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 29 M 2024-04-22T06:18:59.993 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:19:00.595 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-19.0.0-3189.g8d82defa.el9.x86_64.rpm 5.7 MB/s | 3.4 MB 00:00 2024-04-22T06:19:00.595 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:19:00.596 INFO:teuthology.orchestra.run.smithi049.stdout:Total 5.6 MB/s | 3.4 MB 00:00 2024-04-22T06:19:00.596 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:19:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:19:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:19:00.701 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:19:00.702 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:19:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:19:01.554 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:19:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:19:01.583 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:19:01.583 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:19:01.583 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:19:01.583 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:19:01.583 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:02.645 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:19:02.645 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:02.645 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:19:02.645 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:19:02.645 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:02.645 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:19:02.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2024-04-22T06:19:03.398 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:35 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:19:03.525 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:19:03.526 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:19:03.526 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 175 k 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 175 k 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 673 k 2024-04-22T06:19:03.527 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:19:03.800 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-19.0.0-3189.g8d82defa.el9.x86_64.rpm 645 kB/s | 175 kB 00:00 2024-04-22T06:19:03.800 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:19:03.801 INFO:teuthology.orchestra.run.smithi049.stdout:Total 641 kB/s | 175 kB 00:00 2024-04-22T06:19:03.801 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:19:03.812 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:19:03.812 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:19:03.880 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:19:03.880 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:19:04.045 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:19:04.131 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:19:04.539 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:19:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:19:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:04.798 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:19:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:19:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:04.798 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:19:05.035 DEBUG:teuthology.parallel:result is None 2024-04-22T06:19:05.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:19:05.368 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:19:05.390 INFO:teuthology.orchestra.run.smithi049.stdout:19.0.0-3189.g8d82defa.el9 2024-04-22T06:19:05.391 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3189.g8d82defa.el9 2024-04-22T06:19:05.391 INFO:teuthology.task.install:The correct ceph version 19.0.0-3189.g8d82defa is installed. 2024-04-22T06:19:05.393 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T06:19:05.393 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:05.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:19:05.455 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T06:19:05.456 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:05.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:19:05.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:19:05.583 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T06:19:05.583 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:05.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:19:05.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:19:05.711 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T06:19:05.711 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:05.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:19:05.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:19:05.842 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T06:19:05.937 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T06:19:05.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2024-04-22T06:19:05.961 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T06:19:05.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T06:19:06.026 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T06:19:06.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T06:19:06.104 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T06:19:06.104 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T06:19:06.104 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188', 'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7668188', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxn98Jkmp/cCnJNWBuK02hv0gAdfj9lrg5UggS4GHSjy1MO13Y9kb/aUfG3A1Co3x2gWHMmKYKh3a/6R3EWvq0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-22_05:57:37', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T06:19:06.105 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T06:19:06.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T06:19:06.210 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:06.211 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:19:06.266 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:19:06.267 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-04-22T06:19:06.321 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:19:06.321 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:19:06.321 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-22T06:19:06.322 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:19:06.322 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:19:06.322 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 06:19:01.842157408 +0000 2024-04-22T06:19:06.322 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 06:15:39.049708922 +0000 2024-04-22T06:19:06.322 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 06:15:39.049708922 +0000 2024-04-22T06:19:06.322 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-04-22 06:15:39.049708922 +0000 2024-04-22T06:19:06.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:19:06.383 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T06:19:06.384 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T06:19:06.384 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000103495 s, 4.9 MB/s 2024-04-22T06:19:06.385 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:19:06.445 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-04-22T06:19:06.500 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 06:19:01.842157408 +0000 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 06:15:39.393699458 +0000 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 06:15:39.393699458 +0000 2024-04-22T06:19:06.501 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-04-22 06:15:39.393699458 +0000 2024-04-22T06:19:06.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:19:06.562 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T06:19:06.563 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T06:19:06.563 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.5551e-05 s, 5.4 MB/s 2024-04-22T06:19:06.564 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:19:06.620 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-04-22T06:19:06.675 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 06:19:01.843157380 +0000 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 06:15:39.733690105 +0000 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 06:15:39.733690105 +0000 2024-04-22T06:19:06.676 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-04-22 06:15:39.733690105 +0000 2024-04-22T06:19:06.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:19:06.738 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T06:19:06.738 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T06:19:06.738 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.00010101 s, 5.1 MB/s 2024-04-22T06:19:06.739 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:19:06.795 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-04-22T06:19:06.852 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:19:06.852 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-04-22 06:19:01.843157380 +0000 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-04-22 06:15:40.062681054 +0000 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-04-22 06:15:40.062681054 +0000 2024-04-22T06:19:06.853 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-04-22 06:15:40.062681054 +0000 2024-04-22T06:19:06.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:19:06.914 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-04-22T06:19:06.915 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-04-22T06:19:06.915 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000101465 s, 5.0 MB/s 2024-04-22T06:19:06.916 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:19:06.973 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:19:06.973 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T06:19:06.973 INFO:tasks.ceph:Generating config... 2024-04-22T06:19:06.974 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-22T06:19:06.974 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T06:19:06.974 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-22T06:19:06.974 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-22T06:19:06.974 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-22T06:19:06.975 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] log to file = False 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T06:19:06.976 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-22T06:19:06.976 INFO:tasks.ceph:[global] auth client required = none 2024-04-22T06:19:06.976 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-22T06:19:06.976 INFO:tasks.ceph:[global] auth service required = none 2024-04-22T06:19:06.976 INFO:tasks.ceph:[global] auth supported = none 2024-04-22T06:19:06.976 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-22T06:19:06.976 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-22T06:19:06.977 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-22T06:19:06.978 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-22T06:19:06.979 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-22T06:19:06.980 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-22T06:19:06.981 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-22T06:19:06.981 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T06:19:06.981 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T06:19:06.981 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-22T06:19:06.981 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-22T06:19:06.982 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-22T06:19:06.982 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-22T06:19:06.982 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-22T06:19:06.982 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-22T06:19:06.982 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-22T06:19:06.982 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-22T06:19:06.982 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T06:19:06.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T06:19:07.065 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T06:19:07.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T06:19:07.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:19:07.141 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.49')] 2024-04-22T06:19:07.142 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.49', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-22T06:19:07.143 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:07.143 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T06:19:07.196 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.49 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:07.288+0000 7f8584869f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:07.288+0000 7f8584869f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:07.288+0000 7f8584869f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid 79260856-af10-47bc-a5d9-58ee0e6222de 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 79260856-af10-47bc-a5d9-58ee0e6222de 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2024-04-22T06:19:07.289652+0000 2024-04-22T06:19:07.289 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-04-22T06:19:07.289652+0000 2024-04-22T06:19:07.290 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 17 (quincy) 2024-04-22T06:19:07.290 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 1 2024-04-22T06:19:07.290 INFO:teuthology.orchestra.run.smithi049.stdout:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-04-22T06:19:07.290 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-22T06:19:07.291 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T06:19:07.306 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 79260856-af10-47bc-a5d9-58ee0e6222de... 2024-04-22T06:19:07.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T06:19:07.387 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = none 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2024-04-22T06:19:07.388 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2024-04-22T06:19:07.389 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2024-04-22T06:19:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.391 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2024-04-22T06:19:07.391 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.391 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = 172.21.15.49 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: auth client required = none 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: auth cluster required = none 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: auth service required = none 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: debug asok = 0/0 2024-04-22T06:19:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug bdev = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug buffer = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug civetweb = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug client = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug compressor = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug context = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug crush = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug crypto = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug dpdk = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug eventtrace = 0/0 2024-04-22T06:19:07.395 INFO:teuthology.orchestra.run.smithi049.stdout: debug filer = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug filestore = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug finisher = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug fuse = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug heartbeatmap = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug javaclient = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug journal = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug journaler = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug kstore = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug leveldb = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug lockdep = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds balancer = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds locker = 0/0 2024-04-22T06:19:07.396 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds log = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds log expire = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds migrator = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug memdb = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgrc = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug objclass = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug objectcacher = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug objecter = 0/0 2024-04-22T06:19:07.397 INFO:teuthology.orchestra.run.smithi049.stdout: debug optracker = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug perfcounter = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rados = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd mirror = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd replay = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug refs = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw lifecycle = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw notification = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw sync = 0/0 2024-04-22T06:19:07.398 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 0/0 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug striper = 0/0 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug throttle = 0/0 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug timer = 0/0 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: debug tp = 0/0 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: ms client mode = crc 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: ms cluster mode = crc 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon client mode = crc 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon cluster mode = crc 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon service mode = crc 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: ms service mode = crc 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 79260856-af10-47bc-a5d9-58ee0e6222de 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.399 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug trim objects = true 2024-04-22T06:19:07.400 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.401 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 20 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore block size = 96636764160 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore compression algorithm = snappy 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore compression mode = aggressive 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore fsck on mount = True 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore rocksdb cf = False 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: crimson osd obc lru size = 10 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 20 2024-04-22T06:19:07.402 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 20 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug filestore = 0/0 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug journal = 0/0 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 0/0 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 10 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: err to stderr = True 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: log flush on exit = True 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: log to file = False 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: log to stderr = True 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd full ratio = 0.9 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd nearfull ratio = 0.8 2024-04-22T06:19:07.403 INFO:teuthology.orchestra.run.smithi049.stdout: osd failsafe full ratio = 0.95 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: osd objectstore = bluestore 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: osd shutdown pgref assert = True 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2024-04-22T06:19:07.404 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 0/0 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 0/0 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T06:19:07.405 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default crimson = True 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2024-04-22T06:19:07.406 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.3] 2024-04-22T06:19:07.407 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:19:07.408 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T06:19:07.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T06:19:07.512 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T06:19:07.513 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:07.513 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T06:19:07.528 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:07.528 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T06:19:07.583 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:19:07.583 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:07.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T06:19:07.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:19:07.657 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:07.657 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:07.712 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T06:19:07.713 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T06:19:07.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T06:19:07.810 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T06:19:07.813 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T06:19:07.813 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T06:19:07.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T06:19:07.861 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T06:19:07.874 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T06:19:07.874 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T06:19:07.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T06:19:07.898 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:19:07.899 INFO:tasks.ceph:role: osd.0 2024-04-22T06:19:07.899 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:19:07.899 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T06:19:07.962 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:19:07.962 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:19:07.962 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:19:07.962 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:19:07.962 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:19:07.963 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:19:07.963 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:19:07.963 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:19:07.963 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:19:07.963 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:19:08.001 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-04-22T06:19:08.002 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-04-22T06:19:08.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T06:19:08.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T06:19:08.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T06:19:08.210 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:19:08.210 INFO:tasks.ceph:role: osd.1 2024-04-22T06:19:08.210 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:19:08.210 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T06:19:08.273 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:19:08.274 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:19:08.313 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-04-22T06:19:08.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-04-22T06:19:08.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T06:19:08.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T06:19:08.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T06:19:08.520 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:19:08.520 INFO:tasks.ceph:role: osd.2 2024-04-22T06:19:08.520 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:19:08.520 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T06:19:08.584 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:19:08.584 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:19:08.584 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:19:08.584 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:19:08.584 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:19:08.585 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:19:08.585 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:19:08.585 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:19:08.585 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:19:08.585 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:19:08.624 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-04-22T06:19:08.625 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-04-22T06:19:08.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T06:19:08.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T06:19:08.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:09.064 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:19:09.065 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-04-22 06:19:09,066 seastar - Reactor backend: linux-aio 2024-04-22T06:19:09.076 INFO:teuthology.orchestra.run.smithi049.stderr:==26257==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:19:09.207 INFO:teuthology.orchestra.run.smithi049.stderr:DEBUG 2024-04-22 06:19:09,208 [shard 0:main] seastar - reactor::drain 2024-04-22T06:19:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-04-22T06:19:09.759 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-04-22T06:19:09.759 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-04-22T06:19:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:09.759 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:09.819 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:19:09.823 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-04-22 06:19:09,824 seastar - Reactor backend: linux-aio 2024-04-22T06:19:09.833 INFO:teuthology.orchestra.run.smithi049.stderr:==26255==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:19:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x36889 0x5db8f5f 0x5db912a 0x5db96b6 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x368158a 0x2958f 0x2963f 0x346d444 2024-04-22T06:19:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2024-04-22T06:19:14.493 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:14.493 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-04-22T06:19:14.493 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-04-22T06:19:14.493 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-04-22T06:19:14.493 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:14.493 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:14.494 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 79260856-af10-47bc-a5d9-58ee0e6222de 2024-04-22T06:19:14.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T06:19:14.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:14.842 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:19:14.843 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-04-22 06:19:14,844 seastar - Reactor backend: linux-aio 2024-04-22T06:19:14.849 INFO:teuthology.orchestra.run.smithi049.stderr:==26439==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:19:14.980 INFO:teuthology.orchestra.run.smithi049.stderr:DEBUG 2024-04-22 06:19:14,981 [shard 0:main] seastar - reactor::drain 2024-04-22T06:19:15.532 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:15.532 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-04-22T06:19:15.532 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-04-22T06:19:15.532 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-04-22T06:19:15.532 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:15.532 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:15.589 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:19:15.593 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-04-22 06:19:15,594 seastar - Reactor backend: linux-aio 2024-04-22T06:19:15.602 INFO:teuthology.orchestra.run.smithi049.stderr:==26437==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:19:20.282 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:20.282 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-04-22T06:19:20.283 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-04-22T06:19:20.283 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-04-22T06:19:20.283 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:20.283 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:20.283 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 79260856-af10-47bc-a5d9-58ee0e6222de 2024-04-22T06:19:20.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T06:19:20.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:20.637 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:19:20.637 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-04-22 06:19:20,638 seastar - Reactor backend: linux-aio 2024-04-22T06:19:20.643 INFO:teuthology.orchestra.run.smithi049.stderr:==26621==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:19:20.747 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x362fa 0x5deb354 0x5deb536 0x5deb74d 0x5deba23 0x5e3a263 0x5e3a469 0x5db8f50 0x5db912a 0x5db96b6 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x37e091a 0x37e1473 0x3680d7a 0x2958f 0x2963f 0x346d444 2024-04-22T06:19:20.747 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2024-04-22T06:19:20.774 INFO:teuthology.orchestra.run.smithi049.stderr:DEBUG 2024-04-22 06:19:20,774 [shard 0:main] seastar - reactor::drain 2024-04-22T06:19:21.326 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:21.326 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-04-22T06:19:21.327 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-04-22T06:19:21.327 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-04-22T06:19:21.327 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:21.327 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:21.384 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:19:21.388 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-04-22 06:19:21,389 seastar - Reactor backend: linux-aio 2024-04-22T06:19:21.397 INFO:teuthology.orchestra.run.smithi049.stderr:==26619==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:19:21.462 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x349062d 0x37ea25d 0x5a0e6a1 0x5a1170f 0x5ce3817 0x5cdda62 0x5bffebb 0x5c0f812 0x5c157b0 0x5c181af 0x5c18dc6 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x368158a 0x2958f 0x2963f 0x346d444 2024-04-22T06:19:21.462 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:26.056 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 79260856-af10-47bc-a5d9-58ee0e6222de 2024-04-22T06:19:26.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T06:19:26.103 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T06:19:26.103 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:26.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T06:19:26.168 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:26.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T06:19:26.231 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:26.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T06:19:26.295 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:26.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T06:19:26.359 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:26.359 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T06:19:26.416 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T06:19:26.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBbASZm+11iMBAAAnYwiE5+NZAQm0uvYPFS1g== 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBdASZmUMjbOhAAAj502BFpSrhLmeMH2YKtMQ== 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBjASZmrU/iLBAAzmPyLMD5JUJcl9JGDvTrHA== 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBpASZmJX+EIBAAA5j9J2yMnHMom0U5fxFSsQ== 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-04-22T06:19:26.479 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBbASZmPT5vMxAABNp8pHanxHyY2qkNrorjNg== 2024-04-22T06:19:26.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T06:19:26.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:19:26.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:19:26.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:19:26.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T06:19:26.821 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T06:19:26.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T06:19:26.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T06:19:26.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:26.965+0000 7f7bf3438a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:26.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:26.965+0000 7f7bf3438a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:26.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T06:19:27.003 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:19:27.057 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T06:19:27.058 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T06:19:27.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T06:19:27.100 INFO:tasks.ceph.mon.a:Started 2024-04-22T06:19:27.100 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T06:19:27.100 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T06:19:27.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T06:19:27.102 INFO:tasks.ceph.mgr.x:Started 2024-04-22T06:19:27.103 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T06:19:27.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T06:19:27.185 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-04-22T06:19:27.186+0000 7f392da2fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.186 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-04-22T06:19:27.186+0000 7f392da2fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.186 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-04-22T06:19:27.186+0000 7f392da2fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.187 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.188+0000 7f7ec0d6d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:27.211+0000 7f6088be4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:27.219+0000 7f6088be4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.291 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-04-22T06:19:27.302 INFO:tasks.ceph:check_enable_crimson: True 2024-04-22T06:19:27.302 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-22T06:19:27.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-22T06:19:27.391 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.391+0000 7f7ec0d6d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.391 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.392+0000 7f7ec0d6d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:27.407+0000 7f0b46ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:27.413+0000 7f0b46ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:27.495 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-04-22T06:19:27.495+0000 7f39243f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-22T06:19:27.495 INFO:tasks.ceph.mon.a.smithi049.stderr:Please be aware that this feature is experimental, untested, 2024-04-22T06:19:27.495 INFO:tasks.ceph.mon.a.smithi049.stderr:unsupported, and may result in data corruption, data loss, 2024-04-22T06:19:27.495 INFO:tasks.ceph.mon.a.smithi049.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-22T06:19:27.495 INFO:tasks.ceph.mon.a.smithi049.stderr:feature with important data. 2024-04-22T06:19:27.495 INFO:tasks.ceph.mon.a.smithi049.stderr: 2024-04-22T06:19:27.505 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.506+0000 7f7ec0d6d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:19:27.675 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.676+0000 7f7ec0d6d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:19:27.731 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.732+0000 7f7ec0d6d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:19:27.991 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:27.991+0000 7f7ec0d6d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:19:28.038 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:28.039+0000 7f7ec0d6d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:19:28.303 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T06:19:28.303 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:28.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T06:19:28.328 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:28.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T06:19:28.391 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:28.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T06:19:28.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 5ecacfb2-cd31-4bee-ba67-c49d46dd686c 0 2024-04-22T06:19:28.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:28.602+0000 7ffa1b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:28.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:28.610+0000 7ffa1b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:28.688 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:28.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 3ed72f93-e067-428a-a647-22885632c60b 1 2024-04-22T06:19:28.789 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:28.789+0000 7f7ec0d6d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:19:28.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:28.808+0000 7fdfc48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:28.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:28.816+0000 7fdfc48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:28.885 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:28.885+0000 7f7ec0d6d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:19:28.897 INFO:teuthology.orchestra.run.smithi049.stdout:1 2024-04-22T06:19:28.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 8ce3e041-85c0-4f9c-b1fa-0521d5be36bf 2 2024-04-22T06:19:28.932 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:28.932+0000 7f7ec0d6d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:19:29.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:29.013+0000 7f6cdee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:29.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:29.020+0000 7f6cdee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:29.101 INFO:teuthology.orchestra.run.smithi049.stdout:2 2024-04-22T06:19:29.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-22T06:19:29.113 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:29.113+0000 7f7ec0d6d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:19:29.143 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T06:19:29.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T06:19:29.185 INFO:tasks.ceph.osd.0:Started 2024-04-22T06:19:29.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-22T06:19:29.186 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:29.186+0000 7f7ec0d6d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:19:29.214 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T06:19:29.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T06:19:29.243 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:29.243+0000 7f7ec0d6d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:19:29.245 INFO:tasks.ceph.osd.1:Started 2024-04-22T06:19:29.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-22T06:19:29.274 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T06:19:29.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T06:19:29.318 INFO:tasks.ceph.osd.2:Started 2024-04-22T06:19:29.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:19:29.377 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:29.377+0000 7f7ec0d6d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:19:29.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:29.441+0000 7f9e1266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:29.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:29.448+0000 7f9e1266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:29.517 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:29.518 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":6,"fsid":"79260856-af10-47bc-a5d9-58ee0e6222de","created":"2024-04-22T06:19:27.205619+0000","modified":"2024-04-22T06:19:29.101383+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T06:19:29.101383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5ecacfb2-cd31-4bee-ba67-c49d46dd686c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ed72f93-e067-428a-a647-22885632c60b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8ce3e041-85c0-4f9c-b1fa-0521d5be36bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:19:29.539 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T06:19:29.539 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T06:19:29.841 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T06:19:29.858 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:29.858+0000 7f7ec0d6d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:19:29.953 INFO:teuthology.misc.health.smithi049.stderr:2024-04-22T06:19:29.944+0000 7f574919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:29.953 INFO:teuthology.misc.health.smithi049.stderr:2024-04-22T06:19:29.950+0000 7f574919d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:30.020 INFO:teuthology.misc.health.smithi049.stdout: 2024-04-22T06:19:30.020 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":6,"fsid":"79260856-af10-47bc-a5d9-58ee0e6222de","created":"2024-04-22T06:19:27.205619+0000","modified":"2024-04-22T06:19:29.101383+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T06:19:29.101383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5ecacfb2-cd31-4bee-ba67-c49d46dd686c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3ed72f93-e067-428a-a647-22885632c60b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8ce3e041-85c0-4f9c-b1fa-0521d5be36bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:19:30.032 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-22T06:19:30.151 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:30.151+0000 7f7ec0d6d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:19:30.210 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:30.210+0000 7f7ec0d6d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:19:30.340 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:30.341+0000 7f7ec0d6d000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:19:30.645 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:30.646+0000 7f7ec0d6d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:19:30.698 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:30.699+0000 7f7ec0d6d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:19:30.871 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:30.871+0000 7f7ec0d6d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:19:31.008 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:31.009+0000 7f7ec0d6d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:19:31.061 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:31.062+0000 7f7ec0d6d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:19:31.136 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:31.136+0000 7f7ec0d6d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:19:31.185 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:31.185+0000 7f7ec0d6d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:19:31.738 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:31.738+0000 7f7ec0d6d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:19:31.953 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:31.951+0000 7f7ec0d6d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:19:32.962 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-04-22T06:19:32.962+0000 7f7e5fea0640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-22T06:19:36.335 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T06:19:36.452 INFO:teuthology.misc.health.smithi049.stderr:2024-04-22T06:19:36.449+0000 7fb3d962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:36.454 INFO:teuthology.misc.health.smithi049.stderr:2024-04-22T06:19:36.455+0000 7fb3d962d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:36.653 INFO:teuthology.misc.health.smithi049.stdout: 2024-04-22T06:19:36.654 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":8,"fsid":"79260856-af10-47bc-a5d9-58ee0e6222de","created":"2024-04-22T06:19:27.205619+0000","modified":"2024-04-22T06:19:34.960539+0000","last_up_change":"2024-04-22T06:19:34.960539+0000","last_in_change":"2024-04-22T06:19:29.101383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"5ecacfb2-cd31-4bee-ba67-c49d46dd686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":195346896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":195346896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":195346896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":195346896}]},"public_addr":"172.21.15.49:6807/195346896","cluster_addr":"172.21.15.49:6808/195346896","heartbeat_back_addr":"172.21.15.49:6809/195346896","heartbeat_front_addr":"172.21.15.49:6812/195346896","state":["exists","up"]},{"osd":1,"uuid":"3ed72f93-e067-428a-a647-22885632c60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":3885266955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":3885266955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":3885266955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3885266955}]},"public_addr":"172.21.15.49:6806/3885266955","cluster_addr":"172.21.15.49:6805/3885266955","heartbeat_back_addr":"172.21.15.49:6811/3885266955","heartbeat_front_addr":"172.21.15.49:6810/3885266955","state":["exists","up"]},{"osd":2,"uuid":"8ce3e041-85c0-4f9c-b1fa-0521d5be36bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":731107670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":731107670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":731107670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":731107670}]},"public_addr":"172.21.15.49:6804/731107670","cluster_addr":"172.21.15.49:6803/731107670","heartbeat_back_addr":"172.21.15.49:6813/731107670","heartbeat_front_addr":"172.21.15.49:6814/731107670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:19:36.664 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-22T06:19:36.665 INFO:tasks.ceph:Creating RBD pool 2024-04-22T06:19:36.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T06:19:36.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:36.770+0000 7f51a215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:36.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:36.779+0000 7f51a215d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:37.968 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2024-04-22T06:19:37.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T06:19:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:38.153+0000 7fd4b4824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:38.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:38.166+0000 7fd4b4824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:38.974 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T06:19:38.995 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T06:19:38.996 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T06:19:38.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T06:19:38.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T06:19:38.996 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T06:19:39.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:39.141+0000 7f9e89417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:39.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:39.146+0000 7f9e89417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:39.399 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:39.411 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":470959499},{"type":"v1","addr":"172.21.15.49:6801","nonce":470959499}]},"active_addr":"172.21.15.49:6801/470959499","active_change":"2024-04-22T06:19:31.953664+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1021269991}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1808406942}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1832657419}]}]} 2024-04-22T06:19:39.415 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T06:19:39.415 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T06:19:39.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:19:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:39.523+0000 7fede7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:39.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:39.529+0000 7fede7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:39.726 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:39.726 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"fsid":"79260856-af10-47bc-a5d9-58ee0e6222de","created":"2024-04-22T06:19:27.205619+0000","modified":"2024-04-22T06:19:38.972832+0000","last_up_change":"2024-04-22T06:19:34.960539+0000","last_in_change":"2024-04-22T06:19:29.101383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:19:35.974609+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T06:19:36.977166+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ecacfb2-cd31-4bee-ba67-c49d46dd686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":195346896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":195346896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":195346896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":195346896}]},"public_addr":"172.21.15.49:6807/195346896","cluster_addr":"172.21.15.49:6808/195346896","heartbeat_back_addr":"172.21.15.49:6809/195346896","heartbeat_front_addr":"172.21.15.49:6812/195346896","state":["exists","up"]},{"osd":1,"uuid":"3ed72f93-e067-428a-a647-22885632c60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":3885266955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":3885266955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":3885266955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3885266955}]},"public_addr":"172.21.15.49:6806/3885266955","cluster_addr":"172.21.15.49:6805/3885266955","heartbeat_back_addr":"172.21.15.49:6811/3885266955","heartbeat_front_addr":"172.21.15.49:6810/3885266955","state":["exists","up"]},{"osd":2,"uuid":"8ce3e041-85c0-4f9c-b1fa-0521d5be36bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":731107670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":731107670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":731107670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":731107670}]},"public_addr":"172.21.15.49:6804/731107670","cluster_addr":"172.21.15.49:6803/731107670","heartbeat_back_addr":"172.21.15.49:6813/731107670","heartbeat_front_addr":"172.21.15.49:6814/731107670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:19:39.739 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T06:19:39.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:19:39.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:39.849+0000 7efc14064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:39.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:39.855+0000 7efc14064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.066 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:40.066 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"fsid":"79260856-af10-47bc-a5d9-58ee0e6222de","created":"2024-04-22T06:19:27.205619+0000","modified":"2024-04-22T06:19:39.976164+0000","last_up_change":"2024-04-22T06:19:34.960539+0000","last_in_change":"2024-04-22T06:19:29.101383+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T06:19:35.974609+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T06:19:36.977166+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5ecacfb2-cd31-4bee-ba67-c49d46dd686c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":195346896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":195346896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":195346896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":195346896}]},"public_addr":"172.21.15.49:6807/195346896","cluster_addr":"172.21.15.49:6808/195346896","heartbeat_back_addr":"172.21.15.49:6809/195346896","heartbeat_front_addr":"172.21.15.49:6812/195346896","state":["exists","up"]},{"osd":1,"uuid":"3ed72f93-e067-428a-a647-22885632c60b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":3885266955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":3885266955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":3885266955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":3885266955}]},"public_addr":"172.21.15.49:6806/3885266955","cluster_addr":"172.21.15.49:6805/3885266955","heartbeat_back_addr":"172.21.15.49:6811/3885266955","heartbeat_front_addr":"172.21.15.49:6810/3885266955","state":["exists","up"]},{"osd":2,"uuid":"8ce3e041-85c0-4f9c-b1fa-0521d5be36bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":731107670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":731107670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":731107670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":731107670}]},"public_addr":"172.21.15.49:6804/731107670","cluster_addr":"172.21.15.49:6803/731107670","heartbeat_back_addr":"172.21.15.49:6813/731107670","heartbeat_front_addr":"172.21.15.49:6814/731107670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:19:40.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T06:19:40.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T06:19:40.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T06:19:40.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.193+0000 7f88d8a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.193+0000 7f7f6aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.193+0000 7fda36d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.200+0000 7f88d8a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.201+0000 7fda36d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.201+0000 7f7f6aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.225 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T06:19:40.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:19:40.229 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T06:19:40.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:19:40.237 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-04-22T06:19:40.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:19:40.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.336+0000 7f79fb754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.343+0000 7f79fb754640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.392+0000 7fcc03ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.394+0000 7f57bb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.399+0000 7fcc03ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:40.401+0000 7f57bb95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:40.546 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:40.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:19:40.600 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:40.605 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:40.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.2 2024-04-22T06:19:40.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T06:19:41.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:19:41.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:19:41.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:19:41.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:41.685+0000 7f035959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:41.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:41.693+0000 7f035959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:41.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:41.737+0000 7f212fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:41.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:41.745+0000 7f212fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:41.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:41.780+0000 7fb75a95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:41.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:41.787+0000 7fb75a95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:41.892 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:41.906 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:19:41.954 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:41.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.2 2024-04-22T06:19:41.991 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:42.004 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T06:19:42.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:19:42.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:19:43.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:19:43.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:43.024+0000 7fcc30cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:43.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:43.031+0000 7fcc30cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:43.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:43.080+0000 7ff4364ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:43.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:43.087+0000 7ff4364ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:43.130+0000 7f0e0df1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:43.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:43.136+0000 7f0e0df1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:43.238 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:43.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:19:43.291 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:43.304 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.2 2024-04-22T06:19:43.338 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:43.351 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T06:19:44.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:19:44.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:19:44.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:19:44.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:44.367+0000 7fbd1b8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:44.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:44.375+0000 7fbd1b8ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:44.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:44.421+0000 7f0eec46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:44.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:44.429+0000 7f0eec46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:44.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:44.464+0000 7f7ae5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:44.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:44.471+0000 7f7ae5e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:44.577 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:44.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:19:44.630 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:44.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.2 2024-04-22T06:19:44.671 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-04-22T06:19:44.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T06:19:45.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:19:45.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:19:45.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:19:45.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:45.710+0000 7f0d7a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:45.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:45.717+0000 7f0d7a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:45.759+0000 7f80d5864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:45.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:45.768+0000 7f80d5864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:45.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:45.800+0000 7faf39e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:45.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:45.807+0000 7faf39e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:45.917 INFO:teuthology.orchestra.run.smithi049.stdout:34359738378 2024-04-22T06:19:45.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738378 for osd.1 2024-04-22T06:19:45.932 DEBUG:teuthology.parallel:result is None 2024-04-22T06:19:45.968 INFO:teuthology.orchestra.run.smithi049.stdout:34359738378 2024-04-22T06:19:45.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738378 for osd.2 2024-04-22T06:19:45.981 DEBUG:teuthology.parallel:result is None 2024-04-22T06:19:46.015 INFO:teuthology.orchestra.run.smithi049.stdout:34359738378 2024-04-22T06:19:46.029 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738378 for osd.0 2024-04-22T06:19:46.029 DEBUG:teuthology.parallel:result is None 2024-04-22T06:19:46.029 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T06:19:46.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:19:46.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:46.176+0000 7fdfd0c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:46.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:46.182+0000 7fdfd0c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:46.374 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T06:19:46.387 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-22T06:19:45.958699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80916,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034604,"statfs":{"total":289910292480,"available":289827434496,"internally_reserved":0,"allocated":430080,"data_stored":1254029,"data_compressed":59828,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.956675"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.029343+0000","last_change":"2024-04-22T06:19:39.093428+0000","last_active":"2024-04-22T06:19:40.029343+0000","last_peered":"2024-04-22T06:19:40.029343+0000","last_clean":"2024-04-22T06:19:40.029343+0000","last_became_active":"2024-04-22T06:19:39.090925+0000","last_became_peered":"2024-04-22T06:19:39.090925+0000","last_unstale":"2024-04-22T06:19:40.029343+0000","last_undegraded":"2024-04-22T06:19:40.029343+0000","last_fullsized":"2024-04-22T06:19:40.029343+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.029930+0000","last_change":"2024-04-22T06:19:39.076633+0000","last_active":"2024-04-22T06:19:40.029930+0000","last_peered":"2024-04-22T06:19:40.029930+0000","last_clean":"2024-04-22T06:19:40.029930+0000","last_became_active":"2024-04-22T06:19:39.074019+0000","last_became_peered":"2024-04-22T06:19:39.074019+0000","last_unstale":"2024-04-22T06:19:40.029930+0000","last_undegraded":"2024-04-22T06:19:40.029930+0000","last_fullsized":"2024-04-22T06:19:40.029930+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.023872+0000","last_change":"2024-04-22T06:19:39.080893+0000","last_active":"2024-04-22T06:19:40.023872+0000","last_peered":"2024-04-22T06:19:40.023872+0000","last_clean":"2024-04-22T06:19:40.023872+0000","last_became_active":"2024-04-22T06:19:39.074965+0000","last_became_peered":"2024-04-22T06:19:39.074965+0000","last_unstale":"2024-04-22T06:19:40.023872+0000","last_undegraded":"2024-04-22T06:19:40.023872+0000","last_fullsized":"2024-04-22T06:19:40.023872+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.029782+0000","last_change":"2024-04-22T06:19:39.089314+0000","last_active":"2024-04-22T06:19:40.029782+0000","last_peered":"2024-04-22T06:19:40.029782+0000","last_clean":"2024-04-22T06:19:40.029782+0000","last_became_active":"2024-04-22T06:19:39.086807+0000","last_became_peered":"2024-04-22T06:19:39.086807+0000","last_unstale":"2024-04-22T06:19:40.029782+0000","last_undegraded":"2024-04-22T06:19:40.029782+0000","last_fullsized":"2024-04-22T06:19:40.029782+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.027683+0000","last_change":"2024-04-22T06:19:39.076288+0000","last_active":"2024-04-22T06:19:40.027683+0000","last_peered":"2024-04-22T06:19:40.027683+0000","last_clean":"2024-04-22T06:19:40.027683+0000","last_became_active":"2024-04-22T06:19:39.073603+0000","last_became_peered":"2024-04-22T06:19:39.073603+0000","last_unstale":"2024-04-22T06:19:40.027683+0000","last_undegraded":"2024-04-22T06:19:40.027683+0000","last_fullsized":"2024-04-22T06:19:40.027683+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.003575+0000","last_change":"2024-04-22T06:19:39.028398+0000","last_active":"2024-04-22T06:19:40.003575+0000","last_peered":"2024-04-22T06:19:40.003575+0000","last_clean":"2024-04-22T06:19:40.003575+0000","last_became_active":"2024-04-22T06:19:39.025928+0000","last_became_peered":"2024-04-22T06:19:39.025928+0000","last_unstale":"2024-04-22T06:19:40.003575+0000","last_undegraded":"2024-04-22T06:19:40.003575+0000","last_fullsized":"2024-04-22T06:19:40.003575+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.002484+0000","last_change":"2024-04-22T06:19:39.055762+0000","last_active":"2024-04-22T06:19:40.002484+0000","last_peered":"2024-04-22T06:19:40.002484+0000","last_clean":"2024-04-22T06:19:40.002484+0000","last_became_active":"2024-04-22T06:19:39.052136+0000","last_became_peered":"2024-04-22T06:19:39.052136+0000","last_unstale":"2024-04-22T06:19:40.002484+0000","last_undegraded":"2024-04-22T06:19:40.002484+0000","last_fullsized":"2024-04-22T06:19:40.002484+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.028146+0000","last_change":"2024-04-22T06:19:39.032071+0000","last_active":"2024-04-22T06:19:40.028146+0000","last_peered":"2024-04-22T06:19:40.028146+0000","last_clean":"2024-04-22T06:19:40.028146+0000","last_became_active":"2024-04-22T06:19:39.029450+0000","last_became_peered":"2024-04-22T06:19:39.029450+0000","last_unstale":"2024-04-22T06:19:40.028146+0000","last_undegraded":"2024-04-22T06:19:40.028146+0000","last_fullsized":"2024-04-22T06:19:40.028146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.025799+0000","last_change":"2024-04-22T06:19:38.061000+0000","last_active":"2024-04-22T06:19:40.025799+0000","last_peered":"2024-04-22T06:19:40.025799+0000","last_clean":"2024-04-22T06:19:40.025799+0000","last_became_active":"2024-04-22T06:19:38.050356+0000","last_became_peered":"2024-04-22T06:19:38.050356+0000","last_unstale":"2024-04-22T06:19:40.025799+0000","last_undegraded":"2024-04-22T06:19:40.025799+0000","last_fullsized":"2024-04-22T06:19:40.025799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:36.965276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:36.965276+0000","last_clean_scrub_stamp":"2024-04-22T06:19:36.965276+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:19:46.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:19:46.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:46.495+0000 7fac5842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:46.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:46.500+0000 7fac5842d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:46.693 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:46.694 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-04-22T06:19:46.705 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-22T06:19:45.958699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80916,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034604,"statfs":{"total":289910292480,"available":289827434496,"internally_reserved":0,"allocated":430080,"data_stored":1254029,"data_compressed":59828,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.956675"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.029343+0000","last_change":"2024-04-22T06:19:39.093428+0000","last_active":"2024-04-22T06:19:40.029343+0000","last_peered":"2024-04-22T06:19:40.029343+0000","last_clean":"2024-04-22T06:19:40.029343+0000","last_became_active":"2024-04-22T06:19:39.090925+0000","last_became_peered":"2024-04-22T06:19:39.090925+0000","last_unstale":"2024-04-22T06:19:40.029343+0000","last_undegraded":"2024-04-22T06:19:40.029343+0000","last_fullsized":"2024-04-22T06:19:40.029343+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.029930+0000","last_change":"2024-04-22T06:19:39.076633+0000","last_active":"2024-04-22T06:19:40.029930+0000","last_peered":"2024-04-22T06:19:40.029930+0000","last_clean":"2024-04-22T06:19:40.029930+0000","last_became_active":"2024-04-22T06:19:39.074019+0000","last_became_peered":"2024-04-22T06:19:39.074019+0000","last_unstale":"2024-04-22T06:19:40.029930+0000","last_undegraded":"2024-04-22T06:19:40.029930+0000","last_fullsized":"2024-04-22T06:19:40.029930+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.023872+0000","last_change":"2024-04-22T06:19:39.080893+0000","last_active":"2024-04-22T06:19:40.023872+0000","last_peered":"2024-04-22T06:19:40.023872+0000","last_clean":"2024-04-22T06:19:40.023872+0000","last_became_active":"2024-04-22T06:19:39.074965+0000","last_became_peered":"2024-04-22T06:19:39.074965+0000","last_unstale":"2024-04-22T06:19:40.023872+0000","last_undegraded":"2024-04-22T06:19:40.023872+0000","last_fullsized":"2024-04-22T06:19:40.023872+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.029782+0000","last_change":"2024-04-22T06:19:39.089314+0000","last_active":"2024-04-22T06:19:40.029782+0000","last_peered":"2024-04-22T06:19:40.029782+0000","last_clean":"2024-04-22T06:19:40.029782+0000","last_became_active":"2024-04-22T06:19:39.086807+0000","last_became_peered":"2024-04-22T06:19:39.086807+0000","last_unstale":"2024-04-22T06:19:40.029782+0000","last_undegraded":"2024-04-22T06:19:40.029782+0000","last_fullsized":"2024-04-22T06:19:40.029782+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.027683+0000","last_change":"2024-04-22T06:19:39.076288+0000","last_active":"2024-04-22T06:19:40.027683+0000","last_peered":"2024-04-22T06:19:40.027683+0000","last_clean":"2024-04-22T06:19:40.027683+0000","last_became_active":"2024-04-22T06:19:39.073603+0000","last_became_peered":"2024-04-22T06:19:39.073603+0000","last_unstale":"2024-04-22T06:19:40.027683+0000","last_undegraded":"2024-04-22T06:19:40.027683+0000","last_fullsized":"2024-04-22T06:19:40.027683+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.003575+0000","last_change":"2024-04-22T06:19:39.028398+0000","last_active":"2024-04-22T06:19:40.003575+0000","last_peered":"2024-04-22T06:19:40.003575+0000","last_clean":"2024-04-22T06:19:40.003575+0000","last_became_active":"2024-04-22T06:19:39.025928+0000","last_became_peered":"2024-04-22T06:19:39.025928+0000","last_unstale":"2024-04-22T06:19:40.003575+0000","last_undegraded":"2024-04-22T06:19:40.003575+0000","last_fullsized":"2024-04-22T06:19:40.003575+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.002484+0000","last_change":"2024-04-22T06:19:39.055762+0000","last_active":"2024-04-22T06:19:40.002484+0000","last_peered":"2024-04-22T06:19:40.002484+0000","last_clean":"2024-04-22T06:19:40.002484+0000","last_became_active":"2024-04-22T06:19:39.052136+0000","last_became_peered":"2024-04-22T06:19:39.052136+0000","last_unstale":"2024-04-22T06:19:40.002484+0000","last_undegraded":"2024-04-22T06:19:40.002484+0000","last_fullsized":"2024-04-22T06:19:40.002484+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.028146+0000","last_change":"2024-04-22T06:19:39.032071+0000","last_active":"2024-04-22T06:19:40.028146+0000","last_peered":"2024-04-22T06:19:40.028146+0000","last_clean":"2024-04-22T06:19:40.028146+0000","last_became_active":"2024-04-22T06:19:39.029450+0000","last_became_peered":"2024-04-22T06:19:39.029450+0000","last_unstale":"2024-04-22T06:19:40.028146+0000","last_undegraded":"2024-04-22T06:19:40.028146+0000","last_fullsized":"2024-04-22T06:19:40.028146+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:37.968138+0000","last_clean_scrub_stamp":"2024-04-22T06:19:37.968138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:19:40.025799+0000","last_change":"2024-04-22T06:19:38.061000+0000","last_active":"2024-04-22T06:19:40.025799+0000","last_peered":"2024-04-22T06:19:40.025799+0000","last_clean":"2024-04-22T06:19:40.025799+0000","last_became_active":"2024-04-22T06:19:38.050356+0000","last_became_peered":"2024-04-22T06:19:38.050356+0000","last_unstale":"2024-04-22T06:19:40.025799+0000","last_undegraded":"2024-04-22T06:19:40.025799+0000","last_fullsized":"2024-04-22T06:19:40.025799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:19:36.965276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:19:36.965276+0000","last_clean_scrub_stamp":"2024-04-22T06:19:36.965276+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:19:46.706 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T06:19:46.706 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T06:19:46.706 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T06:19:46.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T06:19:46.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:46.816+0000 7fa2be2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:46.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:19:46.820+0000 7fa2be2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:19:47.057 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:47.058 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T06:19:47.071 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T06:19:47.071 INFO:teuthology.run_tasks:Running task cbt... 2024-04-22T06:19:47.087 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi049.front.sepia.ceph.com', 'osds': ['ubuntu@smithi049.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi049.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi049.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi049.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-22T06:19:47.087 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-22T06:19:47.110 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-04-22T06:19:47.111 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-22T06:19:47.159 DEBUG:teuthology.orchestra.run.smithi049:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-22T06:19:47.215 INFO:teuthology.orchestra.run.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-22T06:19:47.603 INFO:tasks.cbt:Installing collectl 2024-04-22T06:19:47.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-22T06:19:47.702 INFO:teuthology.orchestra.run.smithi049.stderr:--2024-04-22 06:19:47-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-22T06:19:47.827 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2024-04-22T06:19:47.837 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-04-22T06:19:48.063 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-22T06:19:48.063 INFO:teuthology.orchestra.run.smithi049.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmJgGEzWcr9gor0YRx1XiN2B0H53iU5pqUULue6jkOuDcYBl1C-3ITR_MH2BAhKnn0z4ECgWQmetjaydzrXrLQHcM7Dw%3D%3D&use_mirror=psychz&r= [following] 2024-04-22T06:19:48.063 INFO:teuthology.orchestra.run.smithi049.stderr:--2024-04-22 06:19:48-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmJgGEzWcr9gor0YRx1XiN2B0H53iU5pqUULue6jkOuDcYBl1C-3ITR_MH2BAhKnn0z4ECgWQmetjaydzrXrLQHcM7Dw%3D%3D&use_mirror=psychz&r= 2024-04-22T06:19:48.063 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-22T06:19:48.131 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-22T06:19:48.392 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-22T06:19:48.392 INFO:teuthology.orchestra.run.smithi049.stderr:Location: https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-22T06:19:48.392 INFO:teuthology.orchestra.run.smithi049.stderr:--2024-04-22 06:19:48-- https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-22T06:19:48.403 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)... 208.87.241.191 2024-04-22T06:19:48.469 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)|208.87.241.191|:443... connected. 2024-04-22T06:19:48.789 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T06:19:48.790 INFO:teuthology.orchestra.run.smithi049.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-22T06:19:48.790 INFO:teuthology.orchestra.run.smithi049.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-22T06:19:48.790 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:48.887 INFO:teuthology.orchestra.run.smithi049.stderr: 0K .......... .......... .......... .......... .......... 8% 513K 1s 2024-04-22T06:19:48.953 INFO:teuthology.orchestra.run.smithi049.stderr: 50K .......... .......... .......... .......... .......... 16% 774K 1s 2024-04-22T06:19:48.953 INFO:teuthology.orchestra.run.smithi049.stderr: 100K .......... .......... .......... .......... .......... 24% 124M 0s 2024-04-22T06:19:48.986 INFO:teuthology.orchestra.run.smithi049.stderr: 150K .......... .......... .......... .......... .......... 32% 1.44M 0s 2024-04-22T06:19:49.023 INFO:teuthology.orchestra.run.smithi049.stderr: 200K .......... .......... .......... .......... .......... 41% 1.60M 0s 2024-04-22T06:19:49.023 INFO:teuthology.orchestra.run.smithi049.stderr: 250K .......... .......... .......... .......... .......... 49% 124M 0s 2024-04-22T06:19:49.023 INFO:teuthology.orchestra.run.smithi049.stderr: 300K .......... .......... .......... .......... .......... 57% 108M 0s 2024-04-22T06:19:49.052 INFO:teuthology.orchestra.run.smithi049.stderr: 350K .......... .......... .......... .......... .......... 65% 1.46M 0s 2024-04-22T06:19:49.052 INFO:teuthology.orchestra.run.smithi049.stderr: 400K .......... .......... .......... .......... .......... 74% 102M 0s 2024-04-22T06:19:49.083 INFO:teuthology.orchestra.run.smithi049.stderr: 450K .......... .......... .......... .......... .......... 82% 1.62M 0s 2024-04-22T06:19:49.084 INFO:teuthology.orchestra.run.smithi049.stderr: 500K .......... .......... .......... .......... .......... 90% 106M 0s 2024-04-22T06:19:49.084 INFO:teuthology.orchestra.run.smithi049.stderr: 550K .......... .......... .......... .......... .......... 98% 113M 0s 2024-04-22T06:19:49.084 INFO:teuthology.orchestra.run.smithi049.stderr: 600K ...... 100% 11.4T=0.3s 2024-04-22T06:19:49.084 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:49.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22 06:19:49 (2.02 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-22T06:19:49.084 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:19:49.122 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/ 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/collectl.conf 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Socket.html 2024-04-22T06:19:49.123 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/colmux.html 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Import.html 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/style.css 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-22T06:19:49.124 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Process.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-22T06:19:49.125 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Export.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Startup.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Performance.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Messages.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Logging.html 2024-04-22T06:19:49.126 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Misc.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Examples.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-22T06:19:49.127 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Playback.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Hello.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Network.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Memory.html 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-22T06:19:49.128 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Disks.html 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/service/ 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/service/collectl.service 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/ 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/misc.ph 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/gexpr.ph 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/statsd.ph 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/envrules.std 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/hello.ph 2024-04-22T06:19:49.129 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vmsum.ph 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/proctree.ph 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/README-WINDOWS 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/formatit.ph 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/colmux 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/INSTALL 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/graphite.ph 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vmstat.ph 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/README 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/ 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/collectl.1 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/colmux.1 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/lexpr.ph 2024-04-22T06:19:49.130 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/client.pl 2024-04-22T06:19:49.131 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/collectl 2024-04-22T06:19:49.131 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/COPYING 2024-04-22T06:19:49.131 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vnet.ph 2024-04-22T06:19:49.131 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/GPL 2024-04-22T06:19:49.131 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/UNINSTALL 2024-04-22T06:19:49.131 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/ARTISTIC 2024-04-22T06:19:49.177 INFO:teuthology.orchestra.run.smithi049.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-22T06:19:49.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-22T06:19:49.732 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:21 ago on Mon 22 Apr 2024 06:16:28 AM UTC. 2024-04-22T06:19:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-22T06:19:49.819 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-22T06:19:49.868 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-04-22T06:19:49.868 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:19:49.868 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-04-22T06:19:49.868 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:19:49.868 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: librbd-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 23 k 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: perf x86_64 5.14.0-437.el9 appstream 2.2 M 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: libradospp-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 30 k 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout:Install 5 Packages 2024-04-22T06:19:49.869 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:49.870 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.4 M 2024-04-22T06:19:49.870 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 11 M 2024-04-22T06:19:49.870 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-04-22T06:19:50.537 INFO:teuthology.orchestra.run.smithi049.stdout:(1/5): libradospp-devel-19.0.0-3189.g8d82defa.e 176 kB/s | 30 kB 00:00 2024-04-22T06:19:50.654 INFO:teuthology.orchestra.run.smithi049.stdout:(2/5): perf-5.14.0-437.el9.x86_64.rpm 7.7 MB/s | 2.2 MB 00:00 2024-04-22T06:19:50.679 INFO:teuthology.orchestra.run.smithi049.stdout:(3/5): librbd-devel-19.0.0-3189.g8d82defa.el9.x 74 kB/s | 23 kB 00:00 2024-04-22T06:19:50.813 INFO:teuthology.orchestra.run.smithi049.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 431 kB/s | 118 kB 00:00 2024-04-22T06:19:50.838 INFO:teuthology.orchestra.run.smithi049.stdout:(5/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 71 kB/s | 13 kB 00:00 2024-04-22T06:19:50.839 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:19:50.839 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.5 MB/s | 2.4 MB 00:00 2024-04-22T06:19:50.869 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-04-22T06:19:50.891 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-04-22T06:19:50.892 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-04-22T06:19:51.007 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-04-22T06:19:51.008 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-04-22T06:19:51.308 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-04-22T06:19:51.382 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-22T06:19:51.596 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-22T06:19:51.711 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradospp-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 3/5 2024-04-22T06:19:52.039 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librbd-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 4/5 2024-04-22T06:19:52.124 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : perf-5.14.0-437.el9.x86_64 5/5 2024-04-22T06:19:52.895 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: perf-5.14.0-437.el9.x86_64 5/5 2024-04-22T06:19:52.895 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradospp-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/5 2024-04-22T06:19:52.895 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2/5 2024-04-22T06:19:52.895 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : perf-5.14.0-437.el9.x86_64 3/5 2024-04-22T06:19:52.895 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: libradospp-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: librbd-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-2.34-7.el9.x86_64 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-22T06:19:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: perf-5.14.0-437.el9.x86_64 2024-04-22T06:19:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:53.180 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-04-22T06:19:53.461 INFO:tasks.cbt:benchmark: librbdfio 2024-04-22T06:19:53.461 DEBUG:teuthology.orchestra.run.smithi049:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-04-22T06:19:53.476 INFO:teuthology.orchestra.run.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-04-22T06:19:55.704 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-04-22T06:19:56.219 INFO:teuthology.orchestra.run.smithi049.stdout:Operating system Linux 2024-04-22T06:19:56.220 INFO:teuthology.orchestra.run.smithi049.stdout:CPU x86_64 2024-04-22T06:19:56.220 INFO:teuthology.orchestra.run.smithi049.stdout:Big endian no 2024-04-22T06:19:56.220 INFO:teuthology.orchestra.run.smithi049.stdout:Compiler gcc 2024-04-22T06:19:56.220 INFO:teuthology.orchestra.run.smithi049.stdout:Cross compile no 2024-04-22T06:19:56.220 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-04-22T06:19:56.220 INFO:teuthology.orchestra.run.smithi049.stdout:Static build no 2024-04-22T06:19:56.316 INFO:teuthology.orchestra.run.smithi049.stdout:Wordsize 64 2024-04-22T06:19:56.323 INFO:teuthology.orchestra.run.smithi049.stdout:zlib no 2024-04-22T06:19:56.357 INFO:teuthology.orchestra.run.smithi049.stdout:fcntl(F_FULLFSYNC) no 2024-04-22T06:19:56.372 INFO:teuthology.orchestra.run.smithi049.stdout:Linux AIO support no 2024-04-22T06:19:56.372 INFO:teuthology.orchestra.run.smithi049.stdout:Linux AIO support rw flags no 2024-04-22T06:19:56.432 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO support yes 2024-04-22T06:19:56.433 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO support needs -lrt no 2024-04-22T06:19:56.464 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO fsync yes 2024-04-22T06:19:56.496 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX pshared support yes 2024-04-22T06:19:56.533 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_condattr_setclock() yes 2024-04-22T06:19:56.574 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_sigmask() yes 2024-04-22T06:19:56.611 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_getaffinity_np() yes 2024-04-22T06:19:56.620 INFO:teuthology.orchestra.run.smithi049.stdout:Solaris AIO support no 2024-04-22T06:19:56.677 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_fetch_and_add yes 2024-04-22T06:19:56.716 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_synchronize yes 2024-04-22T06:19:56.746 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_val_compare_and_swap yes 2024-04-22T06:19:56.753 INFO:teuthology.orchestra.run.smithi049.stdout:libverbs no 2024-04-22T06:19:56.767 INFO:teuthology.orchestra.run.smithi049.stdout:rdmacm no 2024-04-22T06:19:56.775 INFO:teuthology.orchestra.run.smithi049.stdout:librpma no 2024-04-22T06:19:56.785 INFO:teuthology.orchestra.run.smithi049.stdout:libprotobuf_c no 2024-04-22T06:19:56.816 INFO:teuthology.orchestra.run.smithi049.stdout:asprintf() yes 2024-04-22T06:19:56.846 INFO:teuthology.orchestra.run.smithi049.stdout:vasprintf() yes 2024-04-22T06:19:56.879 INFO:teuthology.orchestra.run.smithi049.stdout:Linux fallocate yes 2024-04-22T06:19:56.910 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX fadvise yes 2024-04-22T06:19:56.943 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX fallocate yes 2024-04-22T06:19:56.973 INFO:teuthology.orchestra.run.smithi049.stdout:sched_setaffinity(3 arg) yes 2024-04-22T06:19:56.973 INFO:teuthology.orchestra.run.smithi049.stdout:sched_setaffinity(2 arg) no 2024-04-22T06:19:57.005 INFO:teuthology.orchestra.run.smithi049.stdout:clock_gettime yes 2024-04-22T06:19:57.037 INFO:teuthology.orchestra.run.smithi049.stdout:CLOCK_MONOTONIC yes 2024-04-22T06:19:57.070 INFO:teuthology.orchestra.run.smithi049.stdout:clockid_t yes 2024-04-22T06:19:57.102 INFO:teuthology.orchestra.run.smithi049.stdout:gettimeofday yes 2024-04-22T06:19:57.135 INFO:teuthology.orchestra.run.smithi049.stdout:fdatasync yes 2024-04-22T06:19:57.165 INFO:teuthology.orchestra.run.smithi049.stdout:pipe() yes 2024-04-22T06:19:57.196 INFO:teuthology.orchestra.run.smithi049.stdout:pipe2() yes 2024-04-22T06:19:57.227 INFO:teuthology.orchestra.run.smithi049.stdout:pread() yes 2024-04-22T06:19:57.322 INFO:teuthology.orchestra.run.smithi049.stdout:sync_file_range yes 2024-04-22T06:19:57.331 INFO:teuthology.orchestra.run.smithi049.stdout:ASharedMemory_create no 2024-04-22T06:19:57.343 INFO:teuthology.orchestra.run.smithi049.stdout:EXT4 move extent yes 2024-04-22T06:19:57.375 INFO:teuthology.orchestra.run.smithi049.stdout:Linux splice(2) yes 2024-04-22T06:19:57.382 INFO:teuthology.orchestra.run.smithi049.stdout:libnuma no 2024-04-22T06:19:57.412 INFO:teuthology.orchestra.run.smithi049.stdout:strsep yes 2024-04-22T06:19:57.444 INFO:teuthology.orchestra.run.smithi049.stdout:strcasestr yes 2024-04-22T06:19:57.459 INFO:teuthology.orchestra.run.smithi049.stdout:strlcat no 2024-04-22T06:19:57.493 INFO:teuthology.orchestra.run.smithi049.stdout:getopt_long_only() yes 2024-04-22T06:19:57.550 INFO:teuthology.orchestra.run.smithi049.stdout:inet_aton yes 2024-04-22T06:19:57.582 INFO:teuthology.orchestra.run.smithi049.stdout:socklen_t yes 2024-04-22T06:19:57.615 INFO:teuthology.orchestra.run.smithi049.stdout:__thread yes 2024-04-22T06:19:57.656 INFO:teuthology.orchestra.run.smithi049.stdout:RUSAGE_THREAD yes 2024-04-22T06:19:57.686 INFO:teuthology.orchestra.run.smithi049.stdout:SCHED_IDLE yes 2024-04-22T06:19:57.721 INFO:teuthology.orchestra.run.smithi049.stdout:TCP_NODELAY yes 2024-04-22T06:19:57.768 INFO:teuthology.orchestra.run.smithi049.stdout:vsock yes 2024-04-22T06:19:57.803 INFO:teuthology.orchestra.run.smithi049.stdout:Net engine window_size yes 2024-04-22T06:19:57.837 INFO:teuthology.orchestra.run.smithi049.stdout:TCP_MAXSEG yes 2024-04-22T06:19:57.868 INFO:teuthology.orchestra.run.smithi049.stdout:RLIMIT_MEMLOCK yes 2024-04-22T06:19:57.902 INFO:teuthology.orchestra.run.smithi049.stdout:pwritev/preadv yes 2024-04-22T06:19:57.935 INFO:teuthology.orchestra.run.smithi049.stdout:pwritev2/preadv2 yes 2024-04-22T06:19:57.973 INFO:teuthology.orchestra.run.smithi049.stdout:IPv6 helpers yes 2024-04-22T06:19:57.990 INFO:teuthology.orchestra.run.smithi049.stdout:http engine no 2024-04-22T06:19:58.135 INFO:teuthology.orchestra.run.smithi049.stdout:Rados engine yes 2024-04-22T06:19:58.294 INFO:teuthology.orchestra.run.smithi049.stdout:Rados Block Device engine yes 2024-04-22T06:19:58.455 INFO:teuthology.orchestra.run.smithi049.stdout:rbd_poll yes 2024-04-22T06:19:58.611 INFO:teuthology.orchestra.run.smithi049.stdout:rbd_invalidate_cache yes 2024-04-22T06:19:58.642 INFO:teuthology.orchestra.run.smithi049.stdout:setvbuf yes 2024-04-22T06:19:58.649 INFO:teuthology.orchestra.run.smithi049.stdout:Gluster API engine no 2024-04-22T06:19:58.665 INFO:teuthology.orchestra.run.smithi049.stdout:s390_z196_facilities no 2024-04-22T06:19:58.665 INFO:teuthology.orchestra.run.smithi049.stdout:HDFS engine no 2024-04-22T06:19:58.717 INFO:teuthology.orchestra.run.smithi049.stdout:MTD yes 2024-04-22T06:19:58.724 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem no 2024-04-22T06:19:58.724 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem1_5 no 2024-04-22T06:19:58.732 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem2 no 2024-04-22T06:19:58.732 INFO:teuthology.orchestra.run.smithi049.stdout:PMDK dev-dax engine no 2024-04-22T06:19:58.732 INFO:teuthology.orchestra.run.smithi049.stdout:PMDK libpmem engine no 2024-04-22T06:19:58.739 INFO:teuthology.orchestra.run.smithi049.stdout:DDN's Infinite Memory Engine no 2024-04-22T06:19:58.739 INFO:teuthology.orchestra.run.smithi049.stdout:iscsi engine no 2024-04-22T06:19:58.739 INFO:teuthology.orchestra.run.smithi049.stdout:NBD engine no 2024-04-22T06:19:58.749 INFO:teuthology.orchestra.run.smithi049.stdout:DAOS File System (dfs) Engine no 2024-04-22T06:19:58.758 INFO:teuthology.orchestra.run.smithi049.stdout:NFS engine no 2024-04-22T06:19:58.758 INFO:teuthology.orchestra.run.smithi049.stdout:lex/yacc for arithmetic no 2024-04-22T06:19:58.791 INFO:teuthology.orchestra.run.smithi049.stdout:getmntent yes 2024-04-22T06:19:58.818 INFO:teuthology.orchestra.run.smithi049.stdout:getmntinfo no 2024-04-22T06:19:58.865 INFO:teuthology.orchestra.run.smithi049.stdout:Static Assert yes 2024-04-22T06:19:58.894 INFO:teuthology.orchestra.run.smithi049.stdout:bool yes 2024-04-22T06:19:58.928 INFO:teuthology.orchestra.run.smithi049.stdout:strndup yes 2024-04-22T06:19:58.936 INFO:teuthology.orchestra.run.smithi049.stdout:Valgrind headers no 2024-04-22T06:19:58.965 INFO:teuthology.orchestra.run.smithi049.stdout:Zoned block device support yes 2024-04-22T06:19:58.994 INFO:teuthology.orchestra.run.smithi049.stdout:Zoned block device capacity yes 2024-04-22T06:19:59.001 INFO:teuthology.orchestra.run.smithi049.stdout:libzbc engine no 2024-04-22T06:19:59.040 INFO:teuthology.orchestra.run.smithi049.stdout:NVMe uring command support yes 2024-04-22T06:19:59.055 INFO:teuthology.orchestra.run.smithi049.stdout:xnvme engine no 2024-04-22T06:19:59.062 INFO:teuthology.orchestra.run.smithi049.stdout:isal no 2024-04-22T06:19:59.078 INFO:teuthology.orchestra.run.smithi049.stdout:libblkio engine no 2024-04-22T06:19:59.078 INFO:teuthology.orchestra.run.smithi049.stdout:march_armv8_a_crc_crypto no 2024-04-22T06:19:59.078 INFO:teuthology.orchestra.run.smithi049.stdout:cuda no 2024-04-22T06:19:59.078 INFO:teuthology.orchestra.run.smithi049.stdout:libcufile no 2024-04-22T06:19:59.110 INFO:teuthology.orchestra.run.smithi049.stdout:Build march=native yes 2024-04-22T06:19:59.114 INFO:teuthology.orchestra.run.smithi049.stdout:CUnit no 2024-04-22T06:19:59.143 INFO:teuthology.orchestra.run.smithi049.stdout:__kernel_rwf_t yes 2024-04-22T06:19:59.171 INFO:teuthology.orchestra.run.smithi049.stdout:-Wimplicit-fallthrough=2 yes 2024-04-22T06:19:59.202 INFO:teuthology.orchestra.run.smithi049.stdout:-Wno-stringop-truncation yes 2024-04-22T06:19:59.240 INFO:teuthology.orchestra.run.smithi049.stdout:MADV_HUGEPAGE yes 2024-04-22T06:19:59.271 INFO:teuthology.orchestra.run.smithi049.stdout:gettid yes 2024-04-22T06:19:59.308 INFO:teuthology.orchestra.run.smithi049.stdout:statx(2)/libc yes 2024-04-22T06:19:59.342 INFO:teuthology.orchestra.run.smithi049.stdout:statx(2)/syscall yes 2024-04-22T06:19:59.344 INFO:teuthology.orchestra.run.smithi049.stdout:Windows PDB generation no 2024-04-22T06:19:59.375 INFO:teuthology.orchestra.run.smithi049.stdout:timerfd_create yes 2024-04-22T06:19:59.377 INFO:teuthology.orchestra.run.smithi049.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-04-22T06:19:59.378 INFO:teuthology.orchestra.run.smithi049.stdout:Lib-based ioengines dynamic no 2024-04-22T06:19:59.442 INFO:teuthology.orchestra.run.smithi049.stdout:TCMalloc support yes 2024-04-22T06:19:59.443 INFO:teuthology.orchestra.run.smithi049.stdout:seed_buckets 4 2024-04-22T06:19:59.527 INFO:teuthology.orchestra.run.smithi049.stderr:FIO_VERSION = fio-3.37-15-gc948 2024-04-22T06:19:59.561 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc16.o 2024-04-22T06:19:59.664 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32.o 2024-04-22T06:19:59.702 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c-arm64.o 2024-04-22T06:19:59.789 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c-intel.o 2024-04-22T06:19:59.836 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c.o 2024-04-22T06:19:59.879 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc64.o 2024-04-22T06:19:59.923 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc7.o 2024-04-22T06:19:59.951 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crct10dif_common.o 2024-04-22T06:19:59.988 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/fnv.o 2024-04-22T06:20:00.038 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/md5.o 2024-04-22T06:20:00.136 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/murmur3.o 2024-04-22T06:20:00.180 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha1.o 2024-04-22T06:20:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha256.o 2024-04-22T06:20:00.691 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha3.o 2024-04-22T06:20:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha512.o 2024-04-22T06:20:00.965 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/test.o 2024-04-22T06:20:01.137 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/xxhash.o 2024-04-22T06:20:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/axmap.o 2024-04-22T06:20:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/bloom.o 2024-04-22T06:20:01.451 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/flist_sort.o 2024-04-22T06:20:01.525 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/gauss.o 2024-04-22T06:20:01.614 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/getrusage.o 2024-04-22T06:20:01.651 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/hweight.o 2024-04-22T06:20:01.691 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/ieee754.o 2024-04-22T06:20:01.742 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/lfsr.o 2024-04-22T06:20:01.847 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/memalign.o 2024-04-22T06:20:01.892 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/memcpy.o 2024-04-22T06:20:02.046 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/mountcheck.o 2024-04-22T06:20:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/num2str.o 2024-04-22T06:20:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/output_buffer.o 2024-04-22T06:20:02.206 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/pattern.o 2024-04-22T06:20:02.356 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/prio_tree.o 2024-04-22T06:20:02.481 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/rand.o 2024-04-22T06:20:02.677 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/rbtree.o 2024-04-22T06:20:02.805 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/strntol.o 2024-04-22T06:20:02.854 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/zipf.o 2024-04-22T06:20:02.979 INFO:teuthology.orchestra.run.smithi049.stdout: CC gettime.o 2024-04-22T06:20:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: CC ioengines.o 2024-04-22T06:20:03.568 INFO:teuthology.orchestra.run.smithi049.stdout: CC init.o 2024-04-22T06:20:04.375 INFO:teuthology.orchestra.run.smithi049.stdout: CC stat.o 2024-04-22T06:20:06.317 INFO:teuthology.orchestra.run.smithi049.stdout: CC log.o 2024-04-22T06:20:06.449 INFO:teuthology.orchestra.run.smithi049.stdout: CC time.o 2024-04-22T06:20:06.577 INFO:teuthology.orchestra.run.smithi049.stdout: CC filesetup.o 2024-04-22T06:20:07.315 INFO:teuthology.orchestra.run.smithi049.stdout: CC eta.o 2024-04-22T06:20:07.651 INFO:teuthology.orchestra.run.smithi049.stdout: CC verify.o 2024-04-22T06:20:08.371 INFO:teuthology.orchestra.run.smithi049.stdout: CC memory.o 2024-04-22T06:20:08.551 INFO:teuthology.orchestra.run.smithi049.stdout: CC io_u.o 2024-04-22T06:20:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: CC parse.o 2024-04-22T06:20:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: CC fio_sem.o 2024-04-22T06:20:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: CC rwlock.o 2024-04-22T06:20:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: CC pshared.o 2024-04-22T06:20:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: CC options.o 2024-04-22T06:20:10.928 INFO:teuthology.orchestra.run.smithi049.stdout: CC smalloc.o 2024-04-22T06:20:11.126 INFO:teuthology.orchestra.run.smithi049.stdout: CC filehash.o 2024-04-22T06:20:11.282 INFO:teuthology.orchestra.run.smithi049.stdout: CC profile.o 2024-04-22T06:20:11.407 INFO:teuthology.orchestra.run.smithi049.stdout: CC debug.o 2024-04-22T06:20:11.450 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/cpu.o 2024-04-22T06:20:11.593 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/mmap.o 2024-04-22T06:20:11.766 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/sync.o 2024-04-22T06:20:12.005 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/null.o 2024-04-22T06:20:12.126 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/net.o 2024-04-22T06:20:12.581 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/ftruncate.o 2024-04-22T06:20:12.685 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/fileoperations.o 2024-04-22T06:20:12.845 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/exec.o 2024-04-22T06:20:13.004 INFO:teuthology.orchestra.run.smithi049.stdout: CC server.o 2024-04-22T06:20:13.774 INFO:teuthology.orchestra.run.smithi049.stdout: CC client.o 2024-04-22T06:20:14.345 INFO:teuthology.orchestra.run.smithi049.stdout: CC iolog.o 2024-04-22T06:20:14.807 INFO:teuthology.orchestra.run.smithi049.stdout: CC backend.o 2024-04-22T06:20:15.697 INFO:teuthology.orchestra.run.smithi049.stdout: CC libfio.o 2024-04-22T06:20:15.883 INFO:teuthology.orchestra.run.smithi049.stdout: CC flow.o 2024-04-22T06:20:16.003 INFO:teuthology.orchestra.run.smithi049.stdout: CC cconv.o 2024-04-22T06:20:16.371 INFO:teuthology.orchestra.run.smithi049.stdout: CC gettime-thread.o 2024-04-22T06:20:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: CC helpers.o 2024-04-22T06:20:16.546 INFO:teuthology.orchestra.run.smithi049.stdout: CC json.o 2024-04-22T06:20:16.735 INFO:teuthology.orchestra.run.smithi049.stdout: CC idletime.o 2024-04-22T06:20:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: CC td_error.o 2024-04-22T06:20:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: CC profiles/tiobench.o 2024-04-22T06:20:17.181 INFO:teuthology.orchestra.run.smithi049.stdout: CC profiles/act.o 2024-04-22T06:20:17.347 INFO:teuthology.orchestra.run.smithi049.stdout: CC io_u_queue.o 2024-04-22T06:20:17.399 INFO:teuthology.orchestra.run.smithi049.stdout: CC filelock.o 2024-04-22T06:20:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: CC workqueue.o 2024-04-22T06:20:17.711 INFO:teuthology.orchestra.run.smithi049.stdout: CC rate-submit.o 2024-04-22T06:20:17.867 INFO:teuthology.orchestra.run.smithi049.stdout: CC optgroup.o 2024-04-22T06:20:17.914 INFO:teuthology.orchestra.run.smithi049.stdout: CC helper_thread.o 2024-04-22T06:20:18.101 INFO:teuthology.orchestra.run.smithi049.stdout: CC steadystate.o 2024-04-22T06:20:18.351 INFO:teuthology.orchestra.run.smithi049.stdout: CC zone-dist.o 2024-04-22T06:20:18.488 INFO:teuthology.orchestra.run.smithi049.stdout: CC zbd.o 2024-04-22T06:20:19.229 INFO:teuthology.orchestra.run.smithi049.stdout: CC dedupe.o 2024-04-22T06:20:19.377 INFO:teuthology.orchestra.run.smithi049.stdout: CC fdp.o 2024-04-22T06:20:19.526 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/posixaio.o 2024-04-22T06:20:19.664 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/falloc.o 2024-04-22T06:20:19.786 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/e4defrag.o 2024-04-22T06:20:19.921 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/splice.o 2024-04-22T06:20:20.115 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/asprintf.o 2024-04-22T06:20:20.153 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strlcat.o 2024-04-22T06:20:20.197 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/mtd.o 2024-04-22T06:20:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/libmtd.o 2024-04-22T06:20:20.777 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/libmtd_legacy.o 2024-04-22T06:20:20.917 INFO:teuthology.orchestra.run.smithi049.stdout: CC diskutil.o 2024-04-22T06:20:21.144 INFO:teuthology.orchestra.run.smithi049.stdout: CC fifo.o 2024-04-22T06:20:21.201 INFO:teuthology.orchestra.run.smithi049.stdout: CC blktrace.o 2024-04-22T06:20:21.548 INFO:teuthology.orchestra.run.smithi049.stdout: CC cgroup.o 2024-04-22T06:20:21.709 INFO:teuthology.orchestra.run.smithi049.stdout: CC trim.o 2024-04-22T06:20:21.831 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/sg.o 2024-04-22T06:20:22.233 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/linux-dev-lookup.o 2024-04-22T06:20:22.292 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/io_uring.o 2024-04-22T06:20:22.694 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/nvme.o 2024-04-22T06:20:23.020 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/linux-blkzoned.o 2024-04-22T06:20:23.180 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/cmdprio.o 2024-04-22T06:20:23.427 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/rados.o 2024-04-22T06:20:23.612 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/rbd.o 2024-04-22T06:20:23.807 INFO:teuthology.orchestra.run.smithi049.stdout: CC fio.o 2024-04-22T06:20:23.899 INFO:teuthology.orchestra.run.smithi049.stdout: LINK fio 2024-04-22T06:20:24.169 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/log.o 2024-04-22T06:20:24.212 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/genzipf.o 2024-04-22T06:20:24.356 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strcasestr.o 2024-04-22T06:20:24.381 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strndup.o 2024-04-22T06:20:24.405 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-genzipf 2024-04-22T06:20:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/btrace2fio.o 2024-04-22T06:20:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-btrace2fio 2024-04-22T06:20:25.133 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/verify-state.o 2024-04-22T06:20:25.234 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/debug.o 2024-04-22T06:20:25.272 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-verify-state 2024-04-22T06:20:25.414 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/stest.o 2024-04-22T06:20:25.485 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/arch.o 2024-04-22T06:20:25.516 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/stest 2024-04-22T06:20:25.664 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/ieee754.o 2024-04-22T06:20:25.706 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/ieee754 2024-04-22T06:20:25.847 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/axmap.o 2024-04-22T06:20:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/axmap 2024-04-22T06:20:26.094 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/lfsr-test.o 2024-04-22T06:20:26.196 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/lfsr-test 2024-04-22T06:20:26.341 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/gen-rand.o 2024-04-22T06:20:26.443 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/gen-rand 2024-04-22T06:20:26.589 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/memlock.o 2024-04-22T06:20:26.652 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/memlock 2024-04-22T06:20:26.795 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/read-to-pipe-async.o 2024-04-22T06:20:26.994 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/read-to-pipe-async 2024-04-22T06:20:27.153 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/io_uring.o 2024-04-22T06:20:27.712 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/io_uring 2024-04-22T06:20:27.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-22T06:20:27.881 DEBUG:teuthology.orchestra.run.smithi049:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-22T06:20:28.114 INFO:teuthology.orchestra.run.smithi049.stderr:06:20:28 - DEBUG - cbt - Settings.cluster: 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'clients': ['ubuntu@smithi049.front.sepia.ceph.com'], 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'head': 'ubuntu@smithi049.front.sepia.ceph.com', 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'is_teuthology': True, 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'iterations': 1, 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'mons': ['ubuntu@smithi049.front.sepia.ceph.com'], 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'osds': ['ubuntu@smithi049.front.sepia.ceph.com'], 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'osds_per_node': 3, 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'pid_dir': '/var/run/ceph', 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'rebuild_every_test': False, 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'rgws': {'ubuntu@smithi049.front.sepia.ceph.com': None}, 2024-04-22T06:20:28.115 INFO:teuthology.orchestra.run.smithi049.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-22T06:20:28.116 INFO:teuthology.orchestra.run.smithi049.stderr: 'use_existing': True, 2024-04-22T06:20:28.116 INFO:teuthology.orchestra.run.smithi049.stderr: 'user': 'ubuntu'} 2024-04-22T06:20:28.116 INFO:teuthology.orchestra.run.smithi049.stderr:06:20:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:20:28.117 INFO:teuthology.orchestra.run.smithi049.stderr:06:20:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:20:28.117 INFO:teuthology.orchestra.run.smithi049.stderr:06:20:28 - DEBUG - cbt - get_fqdn_local()=smithi049 2024-04-22T06:20:28.117 INFO:teuthology.orchestra.run.smithi049.stderr:06:20:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@smithi049.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi 2024-04-22T06:20:28.248 INFO:teuthology.orchestra.run.smithi049.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-22T06:20:28.249 INFO:teuthology.orchestra.run.smithi049.stderr:Traceback (most recent call last): 2024-04-22T06:20:28.249 INFO:teuthology.orchestra.run.smithi049.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 93, in 2024-04-22T06:20:28.249 INFO:teuthology.orchestra.run.smithi049.stderr: exit(main(sys.argv)) 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 64, in main 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: b.initialize() 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/librbdfio.py", line 73, in initialize 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: common.clean_remote_dir(self.run_dir) 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: File "/home/ubuntu/cephtest/cbt/common.py", line 232, in clean_remote_dir 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: pdsh(nodes, 'if [ -d "%s" ]; then rm -rf %s; fi' % (remote_dir, remote_dir), 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: File "/home/ubuntu/cephtest/cbt/common.py", line 55, in communicate 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: raise Exception('\n'.join([str(self), 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr:Exception: checked_Popen args=pdsh -S -f 1 -R ssh -w ubuntu@smithi049.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi continue_if_error=False rtncode=255 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr:stdout: 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr:stderr: 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr:smithi049: Warning: Permanently added 'smithi049.front.sepia.ceph.com' (ECDSA) to the list of known hosts. 2024-04-22T06:20:28.250 INFO:teuthology.orchestra.run.smithi049.stderr:smithi049: Permission denied, please try again. 2024-04-22T06:20:28.251 INFO:teuthology.orchestra.run.smithi049.stderr:smithi049: Permission denied, please try again. 2024-04-22T06:20:28.251 INFO:teuthology.orchestra.run.smithi049.stderr:smithi049: ubuntu@smithi049.front.sepia.ceph.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). 2024-04-22T06:20:28.251 INFO:teuthology.orchestra.run.smithi049.stderr:pdsh@smithi049: smithi049: ssh exited with exit code 255 2024-04-22T06:20:28.251 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:20:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:28.261 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/cbt.py", line 244, in begin self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 2024-04-22T06:20:28.482 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=582bfc9e0129441d8b37957a8f0c7173 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/cbt.py", line 244, in begin self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 2024-04-22T06:20:28.484 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-22T06:20:28.495 DEBUG:teuthology.orchestra.run.smithi049:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-22T06:20:28.533 DEBUG:teuthology.orchestra.run.smithi049:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-04-22T06:20:28.606 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-22T06:20:28.606 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-22T06:20:28.607 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-22T06:20:28.662 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T06:20:28.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T06:20:28.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:20:28.807+0000 7ff502e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:20:28.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-04-22T06:20:28.816+0000 7ff502e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:20:29.031 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T06:20:29.031 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T06:20:29.032 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T06:20:29.032 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:20:29.088 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T06:20:29.089 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T06:20:29.089 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:20:29.147 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T06:20:29.148 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T06:20:29.148 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:20:29.203 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T06:20:29.203 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T06:20:29.203 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T06:20:29.203 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:20:29.229 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T06:20:29.229 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T06:20:29.229 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T06:20:29.229 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:20:29.280 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T06:20:29.280 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T06:20:29.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-22T06:20:29.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:20:29.352 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T06:20:29.553 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:20:29.553 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T06:20:29.633 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:20:29.633 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T06:20:29.716 INFO:tasks.ceph:Archiving mon data... 2024-04-22T06:20:29.718 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188/data/mon.a.tgz 2024-04-22T06:20:29.718 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-04-22T06:20:29.732 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.s0A89sUHEQ 2024-04-22T06:20:29.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.s0A89sUHEQ 2024-04-22T06:20:29.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.s0A89sUHEQ 2024-04-22T06:20:29.903 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.s0A89sUHEQ is 136KB 2024-04-22T06:20:29.956 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.s0A89sUHEQ 2024-04-22T06:20:29.971 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T06:20:29.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T06:20:30.089 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T06:20:30.090 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T06:20:30.092 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188/remote/smithi049/crash 2024-04-22T06:20:30.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T06:20:30.153 INFO:tasks.ceph:Compressing logs... 2024-04-22T06:20:30.153 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:20:30.236 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28804.log 2024-04-22T06:20:30.236 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28972.log 2024-04-22T06:20:30.236 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28606.log 2024-04-22T06:20:30.236 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28804.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28804.log.gz 2024-04-22T06:20:30.236 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28466.log 2024-04-22T06:20:30.236 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28972.log.gz 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28606.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28606.log.gz 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27619.log 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28466.log.gz 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28338.log 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr: 83.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27619.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-22T06:20:30.237 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27619.log.gz 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28338.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28338.log.gz 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28552.log 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28071.log 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28738.log 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28552.log.gz 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr: 86.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-22T06:20:30.238 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28824.log 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28071.log.gz 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32524.log 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28738.log.gz 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29140.log 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28824.log.gz 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.32524.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32524.log.gz 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28339.log 2024-04-22T06:20:30.239 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29140.log.gz 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27344.log 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28339.log.gz 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27344.log.gz 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28426.log 2024-04-22T06:20:30.240 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29056.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27230.log 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28426.log.gz 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27391.log 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27230.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27230.log.gz 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr: /var/log/ceph/ceph-client.admin.28174.log 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27391.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27391.log.gz 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.25783.log 2024-04-22T06:20:30.241 INFO:teuthology.orchestra.run.smithi049.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28930.log 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28174.log.gz 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.tmp-client.admin.25783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28572.log 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25783.log.gz 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28930.log: gzip -5 --verbose -- 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28930.log.gz 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr: /var/log/ceph/ceph-client.admin.28864.log 2024-04-22T06:20:30.242 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28572.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28572.log.gz 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28113.log 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28864.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28864.log.gz 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28698.log 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28113.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28113.log.gz 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28340.log 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28698.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28698.log.gz 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28950.log 2024-04-22T06:20:30.243 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28678.log 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28340.log.gz 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27589.log 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28950.log.gz 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28678.log.gz 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27589.log: gzip -5 --verbose -- 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27589.log.gz 2024-04-22T06:20:30.244 INFO:teuthology.orchestra.run.smithi049.stderr: /var/log/ceph/ceph-client.admin.27431.log 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28216.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27142.log 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27431.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27431.log.gz 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29098.log 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.27142.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27142.log.gz 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28015.log 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29098.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29098.log.gzgzip 2024-04-22T06:20:30.245 INFO:teuthology.orchestra.run.smithi049.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28465.log 2024-04-22T06:20:30.246 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28015.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28015.log.gz 2024-04-22T06:20:30.246 INFO:teuthology.orchestra.run.smithi049.stderr: /var/log/ceph/ceph-client.admin.28258.log 2024-04-22T06:20:30.246 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28465.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28465.log.gz 2024-04-22T06:20:30.246 INFO:teuthology.orchestra.run.smithi049.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-22T06:20:30.246 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.28258.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28258.log.gz 2024-04-22T06:20:30.246 INFO:teuthology.orchestra.run.smithi049.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-22T06:20:30.247 INFO:teuthology.orchestra.run.smithi049.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-22T06:20:30.247 INFO:teuthology.orchestra.run.smithi049.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-22T06:20:30.249 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-04-22T06:20:30.249 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m0.041s 2024-04-22T06:20:30.249 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m0.095s 2024-04-22T06:20:30.249 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.031s 2024-04-22T06:20:30.249 INFO:tasks.ceph:Archiving logs... 2024-04-22T06:20:30.249 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188/remote/smithi049/log 2024-04-22T06:20:30.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T06:20:30.392 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T06:20:30.405 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T06:20:30.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T06:20:30.462 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T06:20:30.472 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T06:20:30.472 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:20:30.486 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:20:30.491 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:20:30.491 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-04-22T06:20:30.491 INFO:teuthology.orchestra.run.smithi049.stdout:^- hv01.front.sepia.ceph.com 2 6 377 7 +1026us[+1024us] +/- 599ms 2024-04-22T06:20:30.491 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 -246us[ -246us] +/- 78ms 2024-04-22T06:20:30.492 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv03.front.sepia.ceph.com 3 6 377 7 +223us[ +221us] +/- 69ms 2024-04-22T06:20:30.492 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:20:30.492 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T06:20:30.544 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T06:20:30.545 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T06:20:30.570 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T06:20:30.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:20:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:30.657 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T06:20:30.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T06:20:30.719 INFO:teuthology.task.internal:Duration was 487.717177 seconds 2024-04-22T06:20:30.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T06:20:30.766 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T06:20:30.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T06:20:30.805 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:20:30.945 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T06:20:30.946 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-04-22T06:20:30.946 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T06:20:31.008 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T06:20:31.009 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T06:20:31.074 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T06:20:31.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T06:20:31.212 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T06:20:31.283 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T06:20:31.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T06:20:31.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T06:20:31.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:20:31.409 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-04-22T06:20:31.448 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:20:31.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:20:31.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T06:20:31.529 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T06:20:31.530 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188/remote/smithi049 2024-04-22T06:20:31.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T06:20:31.580 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T06:20:31.580 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T06:20:31.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T06:20:31.640 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T06:20:31.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T06:20:31.678 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T06:20:31.679 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T06:20:31.693 INFO:teuthology.orchestra.run.smithi049.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 06:20 /home/ubuntu/cephtest 2024-04-22T06:20:31.695 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T06:20:31.712 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T06:20:31.783 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T06:20:31.806 DEBUG:teuthology.nuke:shortname: smithi049 2024-04-22T06:20:31.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:20:31.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668188', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:05:04.655801', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHxn98Jkmp/cCnJNWBuK02hv0gAdfj9lrg5UggS4GHSjy1MO13Y9kb/aUfG3A1Co3x2gWHMmKYKh3a/6R3EWvq0='} 2024-04-22T06:20:31.889 INFO:teuthology.orchestra.console.smithi049:Power off 2024-04-22T06:20:31.890 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T06:20:31.940 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2024-04-22T06:20:31.940 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:20:31.974 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-22T06:20:35.976 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:20:36.086 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-22T06:20:40.088 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:20:40.199 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-04-22T06:20:40.300 INFO:teuthology.orchestra.console.smithi049:Power off completed 2024-04-22T06:20:40.408 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} duration: 487.71717739105225 failure_reason: 'Command failed on smithi049 with status 1: ''/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=582bfc9e0129441d8b37957a8f0c7173 status: fail success: false 2024-04-22T06:20:40.409 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:20:40.511 INFO:teuthology.run:FAIL