2024-04-22T06:09:54.928 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T06:09:54.928 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184 --owner scheduled_yingxin@teuthology --name yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} --verbose -- /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184/orig.config.yaml 2024-04-22T06:09:54.976 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:09:55.115 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184 branch: ci-yingxin-crimson-fix-tri-mutex-4-base description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7668184' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-fix-tri-mutex-4-base ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-fix-tri-mutex-4-base env: CRIMSON_COMPAT: '1' sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-fix-tri-mutex-4-base suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA0D68EFUFtLZRcE4tO2yC1Fv9tXMjfIvB1Rmx18Iz6fjurTGkPY1y6+HtXjK9pqji3SiR9CmYnGRGG68D1oNo= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_05:57:37 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T06:09:55.116 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa; will attempt to use it 2024-04-22T06:09:55.116 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks 2024-04-22T06:09:55.116 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T06:09:55.118 INFO:teuthology.task.internal:Checking packages... 2024-04-22T06:09:55.143 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '8d82defa854b9197ed2c2b855ac5e996c480ec88' 2024-04-22T06:09:55.143 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T06:09:55.143 INFO:teuthology.packaging:ref: None 2024-04-22T06:09:55.143 INFO:teuthology.packaging:tag: None 2024-04-22T06:09:55.143 INFO:teuthology.packaging:branch: ci-yingxin-crimson-fix-tri-mutex-4-base 2024-04-22T06:09:55.143 INFO:teuthology.packaging:sha1: 8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:09:55.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-fix-tri-mutex-4-base 2024-04-22T06:09:55.362 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3189.g8d82defa 2024-04-22T06:09:55.363 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T06:09:55.369 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T06:09:55.369 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T06:09:55.375 INFO:teuthology.task.internal:Saving configuration 2024-04-22T06:09:55.392 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T06:09:55.397 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:09:55.424 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:04:53.289752', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA0D68EFUFtLZRcE4tO2yC1Fv9tXMjfIvB1Rmx18Iz6fjurTGkPY1y6+HtXjK9pqji3SiR9CmYnGRGG68D1oNo='} 2024-04-22T06:09:55.424 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T06:09:55.430 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-22T06:09:55.430 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T06:09:55.485 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcdcb079e50>, signals=[15]) 2024-04-22T06:09:55.485 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T06:09:55.491 INFO:teuthology.task.internal:Opening connections... 2024-04-22T06:09:55.491 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:09:55.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:09:55.571 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T06:09:55.577 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-04-22T06:09:55.651 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-04-22T06:09:55.651 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-04-22T06:09:55.708 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-04-22T06:09:55.708 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-04-22T06:09:55.708 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-04-22T06:09:55.708 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-04-22T06:09:55.708 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-04-22T06:09:55.708 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T06:09:55.709 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T06:09:55.710 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-04-22T06:09:55.736 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T06:09:55.744 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T06:09:55.749 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T06:09:55.750 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-04-22T06:09:55.766 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T06:09:55.771 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T06:09:55.772 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-04-22T06:09:55.830 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T06:09:55.862 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T06:09:55.877 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-22T06:09:55.878 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-22T06:09:55.878 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T06:09:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:09:55.894 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-04-22T06:09:55.950 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-437.el9.x86_64 2024-04-22T06:09:55.950 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-437.el9.x86_64 2024-04-22T06:09:55.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-04-22T06:09:58.187 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-22T06:10:00.013 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-22T06:10:04.497 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.2 MB 00:00 2024-04-22T06:10:06.131 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-04-22T06:10:07.235 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-04-22T06:10:12.265 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-22T06:10:14.088 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T06:10:14.089 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T06:10:14.192 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:10:14.192 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:10:14.193 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:10:14.301 DEBUG:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2024-04-22T06:10:14.804 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 06:10:12 AM UTC. 2024-04-22T06:10:14.922 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:10:14.929 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout:Reinstalling: 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:10:14.930 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 255 k 2024-04-22T06:10:14.931 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 0 2024-04-22T06:10:14.931 INFO:teuthology.orchestra.run.smithi172.stderr:Operation aborted. 2024-04-22T06:10:14.990 INFO:teuthology.orchestra.run.smithi172.stdout:Is this ok [y/N]: 2024-04-22T06:10:14.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2024-04-22T06:10:15.491 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 06:10:12 AM UTC. 2024-04-22T06:10:15.608 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:Reinstalling: 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 255 k 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 0 2024-04-22T06:10:15.609 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:10:16.201 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-437.el9.x86_64.rpm 800 kB/s | 255 kB 00:00 2024-04-22T06:10:16.201 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:10:16.201 INFO:teuthology.orchestra.run.smithi172.stdout:Total 431 kB/s | 255 kB 00:00 2024-04-22T06:10:16.220 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:10:16.409 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:10:16.410 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:10:16.414 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:10:16.414 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:10:16.564 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:10:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:10:16.635 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:10:17.385 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:10:17.386 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T06:10:18.086 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T06:10:18.087 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:10:18.087 INFO:teuthology.orchestra.run.smithi172.stdout:Reinstalled: 2024-04-22T06:10:18.087 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T06:10:18.087 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:10:18.087 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:10:18.254 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T06:10:18.301 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T06:10:18.301 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T06:10:18.301 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T06:10:18.301 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T06:10:18.302 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T06:10:18.333 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-04-22T06:10:18.353 DEBUG:teuthology.parallel:result is None 2024-04-22T06:10:18.353 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T06:10:18.360 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T06:10:18.361 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T06:10:18.408 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T06:10:18.416 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T06:10:18.428 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T06:10:18.428 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T06:10:18.475 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T06:10:18.483 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T06:10:18.483 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T06:10:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:10:18.597 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T06:10:18.599 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T06:10:18.608 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T06:10:18.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T06:10:18.632 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T06:10:18.642 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T06:10:18.642 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T06:10:18.687 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:10:18.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T06:10:18.817 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:10:18.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T06:10:18.939 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:10:18.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T06:10:19.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-04-22T06:10:19.073 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:10:19.526 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T06:10:19.534 INFO:teuthology.task.internal:Starting timer... 2024-04-22T06:10:19.534 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T06:10:19.546 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T06:10:19.559 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-22T06:10:19.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-04-22T06:10:19.623 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-04-22T06:10:19.625 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T06:10:19.625 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-04-22T06:10:19.647 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-04-22T06:10:19.647 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-04-22T06:10:19.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:10:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:10:19.711 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T06:10:19.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-04-22T06:10:19.783 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T06:10:19.793 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-22T06:10:19.798 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-22T06:10:20.305 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T06:10:20.338 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T06:10:20.340 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T06:13:47.015 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-04-22T06:13:47.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-22T06:13:47.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T06:13:47.095 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-04-22T06:13:47.177 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-04-22T06:13:47.178 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T06:13:47.189 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T06:13:47.189 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T06:13:47.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:13:47.246 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T06:13:47.260 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T06:13:47.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-04-22T06:13:47.294 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-04-22T06:13:47.307 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T06:13:47.320 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T06:13:47.376 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:13:47.379 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:13:47.379 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-04-22T06:13:47.379 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:13:47.379 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:13:47.379 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:13:47.380 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:13:47.380 INFO:teuthology.run_tasks:Running task install... 2024-04-22T06:13:47.389 DEBUG:teuthology.task.install:project ceph 2024-04-22T06:13:47.389 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}} 2024-04-22T06:13:47.389 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'} 2024-04-22T06:13:47.389 INFO:teuthology.task.install:Using flavor: crimson 2024-04-22T06:13:47.395 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T06:13:47.395 INFO:teuthology.task.install:extra packages: [] 2024-04-22T06:13:47.395 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'tag': None, 'wait_for_package': False} 2024-04-22T06:13:47.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:13:47.616 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/ 2024-04-22T06:13:47.616 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3189.g8d82defa 2024-04-22T06:13:47.790 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-4-base/8d82defa854b9197ed2c2b855ac5e996c480ec88/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T06:13:47.790 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:13:47.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T06:13:47.816 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T06:13:47.816 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8d82defa854b9197ed2c2b855ac5e996c480ec88/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T06:13:47.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T06:13:47.955 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T06:13:48.019 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-04-22T06:13:48.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-04-22T06:13:48.247 INFO:teuthology.orchestra.run.smithi172.stdout:48 files removed 2024-04-22T06:13:48.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2024-04-22T06:13:48.942 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 215 kB/s | 87 kB 00:00 2024-04-22T06:13:49.368 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-22T06:13:49.693 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 7.0 kB/s | 2.0 kB 00:00 2024-04-22T06:13:50.286 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-22T06:13:52.431 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-22T06:13:56.671 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.2 MB 00:00 2024-04-22T06:13:58.464 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-22T06:14:00.331 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-22T06:14:05.398 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-22T06:14:07.556 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:14:07.558 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:14:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:14:07.558 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:14:07.558 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 10 M 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.7 M 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.9 M 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 19 M 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 25 k 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 796 k 2024-04-22T06:14:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 547 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.3 M 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 45 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 135 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 159 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 315 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 293 k 2024-04-22T06:14:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 99 k 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout:Installing weak dependencies: 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout:Install 30 Packages 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2024-04-22T06:14:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:07.562 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 59 M 2024-04-22T06:14:07.562 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:14:08.399 INFO:teuthology.orchestra.run.smithi172.stdout:(1/32): ceph-base-19.0.0-3189.g8d82defa.el9.x86 11 MB/s | 5.9 MB 00:00 2024-04-22T06:14:08.441 INFO:teuthology.orchestra.run.smithi172.stdout:(2/32): ceph-selinux-19.0.0-3189.g8d82defa.el9. 606 kB/s | 25 kB 00:00 2024-04-22T06:14:08.524 INFO:teuthology.orchestra.run.smithi172.stdout:(3/32): libcephfs2-19.0.0-3189.g8d82defa.el9.x8 9.3 MB/s | 796 kB 00:00 2024-04-22T06:14:08.583 INFO:teuthology.orchestra.run.smithi172.stdout:(4/32): libradosstriper1-19.0.0-3189.g8d82defa. 9.2 MB/s | 547 kB 00:00 2024-04-22T06:14:08.883 INFO:teuthology.orchestra.run.smithi172.stdout:(5/32): librgw2-19.0.0-3189.g8d82defa.el9.x86_6 18 MB/s | 5.3 MB 00:00 2024-04-22T06:14:08.925 INFO:teuthology.orchestra.run.smithi172.stdout:(6/32): python3-ceph-argparse-19.0.0-3189.g8d82 1.1 MB/s | 45 kB 00:00 2024-04-22T06:14:08.975 INFO:teuthology.orchestra.run.smithi172.stdout:(7/32): python3-ceph-common-19.0.0-3189.g8d82de 2.6 MB/s | 135 kB 00:00 2024-04-22T06:14:09.034 INFO:teuthology.orchestra.run.smithi172.stdout:(8/32): python3-cephfs-19.0.0-3189.g8d82defa.el 2.7 MB/s | 159 kB 00:00 2024-04-22T06:14:09.101 INFO:teuthology.orchestra.run.smithi172.stdout:(9/32): python3-rados-19.0.0-3189.g8d82defa.el9 4.6 MB/s | 315 kB 00:00 2024-04-22T06:14:09.151 INFO:teuthology.orchestra.run.smithi172.stdout:(10/32): python3-rbd-19.0.0-3189.g8d82defa.el9. 5.7 MB/s | 293 kB 00:00 2024-04-22T06:14:09.201 INFO:teuthology.orchestra.run.smithi172.stdout:(11/32): python3-rgw-19.0.0-3189.g8d82defa.el9. 1.9 MB/s | 99 kB 00:00 2024-04-22T06:14:09.402 INFO:teuthology.orchestra.run.smithi172.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 535 kB/s | 107 kB 00:00 2024-04-22T06:14:09.477 INFO:teuthology.orchestra.run.smithi172.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-22T06:14:09.535 INFO:teuthology.orchestra.run.smithi172.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-04-22T06:14:09.619 INFO:teuthology.orchestra.run.smithi172.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-04-22T06:14:09.686 INFO:teuthology.orchestra.run.smithi172.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-22T06:14:09.745 INFO:teuthology.orchestra.run.smithi172.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 725 kB/s | 42 kB 00:00 2024-04-22T06:14:09.828 INFO:teuthology.orchestra.run.smithi172.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-22T06:14:09.887 INFO:teuthology.orchestra.run.smithi172.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 616 kB/s | 36 kB 00:00 2024-04-22T06:14:10.138 INFO:teuthology.orchestra.run.smithi172.stdout:(20/32): ceph-common-19.0.0-3189.g8d82defa.el9. 8.5 MB/s | 19 MB 00:02 2024-04-22T06:14:10.154 INFO:teuthology.orchestra.run.smithi172.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 178 kB/s | 47 kB 00:00 2024-04-22T06:14:10.599 INFO:teuthology.orchestra.run.smithi172.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 671 kB/s | 309 kB 00:00 2024-04-22T06:14:10.701 INFO:teuthology.orchestra.run.smithi172.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-22T06:14:10.791 INFO:teuthology.orchestra.run.smithi172.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 527 kB/s | 49 kB 00:00 2024-04-22T06:14:10.891 INFO:teuthology.orchestra.run.smithi172.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-22T06:14:11.025 INFO:teuthology.orchestra.run.smithi172.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-04-22T06:14:11.125 INFO:teuthology.orchestra.run.smithi172.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-22T06:14:11.151 INFO:teuthology.orchestra.run.smithi172.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-04-22T06:14:11.276 INFO:teuthology.orchestra.run.smithi172.stdout:(29/32): ceph-radosgw-19.0.0-3189.g8d82defa.el9 2.9 MB/s | 10 MB 00:03 2024-04-22T06:14:11.318 INFO:teuthology.orchestra.run.smithi172.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-04-22T06:14:11.501 INFO:teuthology.orchestra.run.smithi172.stdout:(31/32): librados2-19.0.0-3189.g8d82defa.el9.x8 9.7 MB/s | 3.4 MB 00:00 2024-04-22T06:14:11.710 INFO:teuthology.orchestra.run.smithi172.stdout:(32/32): librbd1-19.0.0-3189.g8d82defa.el9.x86_ 8.5 MB/s | 3.7 MB 00:00 2024-04-22T06:14:11.713 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:11.714 INFO:teuthology.orchestra.run.smithi172.stdout:Total 14 MB/s | 59 MB 00:04 2024-04-22T06:14:11.816 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:14:11.853 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:14:11.853 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:14:12.313 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:14:12.313 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:14:13.807 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:14:14.322 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T06:14:14.370 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:14:14.462 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:14:14.550 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:14:14.588 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:14:14.634 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:14:14.672 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T06:14:14.701 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T06:14:14.727 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T06:14:14.750 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:14:14.770 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T06:14:14.966 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:14:14.999 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T06:14:15.357 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T06:14:15.420 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:14:15.471 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:14:15.512 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 12/34 2024-04-22T06:14:15.567 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:14:15.593 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 14/34 2024-04-22T06:14:15.670 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T06:14:15.698 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:14:15.786 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 16/34 2024-04-22T06:14:15.813 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T06:14:15.922 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T06:14:15.956 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T06:14:15.996 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T06:14:16.021 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T06:14:16.051 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T06:14:16.081 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T06:14:16.118 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T06:14:16.708 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T06:14:16.759 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:14:16.834 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 26/34 2024-04-22T06:14:16.905 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 27/34 2024-04-22T06:14:16.949 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 28/34 2024-04-22T06:14:19.366 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:14:19.404 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:14:20.072 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 29/34 2024-04-22T06:14:20.115 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:14:20.216 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 30/34 2024-04-22T06:14:20.216 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T06:14:20.216 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T06:14:20.216 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:20.241 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:14:29.926 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:14:29.926 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-04-22T06:14:29.926 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-04-22T06:14:29.926 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-04-22T06:14:29.926 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-04-22T06:14:29.926 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-04-22T06:14:29.927 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-04-22T06:14:29.927 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-04-22T06:14:29.927 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:31.001 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:14:31.025 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 32/34 2024-04-22T06:14:31.025 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:14:31.025 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T06:14:31.025 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:14:31.025 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T06:14:31.026 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:31.027 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:14:31.120 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T06:14:31.120 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:14:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:14:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 1/34 2024-04-22T06:14:32.387 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 3/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 4/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 5/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_6 6/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 7/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9. 8/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x8 9/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 10/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 11/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 12/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 13/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T06:14:32.388 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T06:14:32.389 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T06:14:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T06:14:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T06:14:32.390 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T06:14:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T06:14:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T06:14:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T06:14:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 31/34 2024-04-22T06:14:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T06:14:32.391 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 33/34 2024-04-22T06:14:33.143 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T06:14:33.144 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.145 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:33.146 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:14:33.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2024-04-22T06:14:33.972 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:28 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:14:34.096 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 84 M 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 173 k 2024-04-22T06:14:34.097 INFO:teuthology.orchestra.run.smithi172.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout:Install 9 Packages 2024-04-22T06:14:34.098 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:14:34.099 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 87 M 2024-04-22T06:14:34.099 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 813 M 2024-04-22T06:14:34.099 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:14:34.512 INFO:teuthology.orchestra.run.smithi172.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 709 kB/s | 94 kB 00:00 2024-04-22T06:14:34.630 INFO:teuthology.orchestra.run.smithi172.stdout:(2/9): libcephsqlite-19.0.0-3189.g8d82defa.el9. 691 kB/s | 173 kB 00:00 2024-04-22T06:14:34.714 INFO:teuthology.orchestra.run.smithi172.stdout:(3/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.3 MB/s | 469 kB 00:00 2024-04-22T06:14:34.765 INFO:teuthology.orchestra.run.smithi172.stdout:(4/9): libubsan-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-04-22T06:14:34.874 INFO:teuthology.orchestra.run.smithi172.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-04-22T06:14:34.925 INFO:teuthology.orchestra.run.smithi172.stdout:(6/9): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-04-22T06:14:34.992 INFO:teuthology.orchestra.run.smithi172.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 542 kB/s | 64 kB 00:00 2024-04-22T06:14:35.085 INFO:teuthology.orchestra.run.smithi172.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 763 kB/s | 122 kB 00:00 2024-04-22T06:14:52.232 INFO:teuthology.orchestra.run.smithi172.stdout:(9/9): ceph-test-19.0.0-3189.g8d82defa.el9.x86_ 4.7 MB/s | 84 MB 00:17 2024-04-22T06:14:52.234 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:14:52.235 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.8 MB/s | 87 MB 00:18 2024-04-22T06:14:52.315 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:14:52.338 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:14:52.339 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:14:52.933 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:14:52.934 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:14:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:14:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T06:14:53.908 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T06:14:53.953 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T06:14:54.001 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T06:14:54.035 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:14:54.060 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:14:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T06:14:54.169 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T06:14:54.196 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:15:05.457 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 8/9 2024-04-22T06:15:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:15:11.742 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 9/9 2024-04-22T06:15:11.742 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 1/9 2024-04-22T06:15:11.742 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2/9 2024-04-22T06:15:11.742 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T06:15:11.742 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T06:15:11.742 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T06:15:11.743 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T06:15:11.743 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T06:15:11.743 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T06:15:12.075 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T06:15:12.075 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:12.076 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:12.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2024-04-22T06:15:12.874 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:07 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:12.997 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 6.5 k 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 2.2 M 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 1.6 M 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 249 k 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 5.0 M 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 71 M 2024-04-22T06:15:13.000 INFO:teuthology.orchestra.run.smithi172.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T06:15:13.001 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T06:15:13.002 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout:Install 36 Packages 2024-04-22T06:15:13.003 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:13.004 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 84 M 2024-04-22T06:15:13.004 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 923 M 2024-04-22T06:15:13.004 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:15:13.714 INFO:teuthology.orchestra.run.smithi172.stdout:(1/36): ceph-19.0.0-3189.g8d82defa.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-04-22T06:15:13.895 INFO:teuthology.orchestra.run.smithi172.stdout:(2/36): ceph-mgr-19.0.0-3189.g8d82defa.el9.x86_ 3.7 MB/s | 1.6 MB 00:00 2024-04-22T06:15:13.937 INFO:teuthology.orchestra.run.smithi172.stdout:(3/36): ceph-mds-19.0.0-3189.g8d82defa.el9.x86_ 4.8 MB/s | 2.2 MB 00:00 2024-04-22T06:15:14.020 INFO:teuthology.orchestra.run.smithi172.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3189.g8d82 2.9 MB/s | 249 kB 00:00 2024-04-22T06:15:14.279 INFO:teuthology.orchestra.run.smithi172.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 980 kB/s | 253 kB 00:00 2024-04-22T06:15:14.437 INFO:teuthology.orchestra.run.smithi172.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 7.7 MB/s | 1.2 MB 00:00 2024-04-22T06:15:14.496 INFO:teuthology.orchestra.run.smithi172.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-22T06:15:14.563 INFO:teuthology.orchestra.run.smithi172.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-04-22T06:15:14.647 INFO:teuthology.orchestra.run.smithi172.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 1.5 MB/s | 125 kB 00:00 2024-04-22T06:15:14.714 INFO:teuthology.orchestra.run.smithi172.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-04-22T06:15:14.956 INFO:teuthology.orchestra.run.smithi172.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 780 kB/s | 188 kB 00:00 2024-04-22T06:15:15.064 INFO:teuthology.orchestra.run.smithi172.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-04-22T06:15:15.115 INFO:teuthology.orchestra.run.smithi172.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-04-22T06:15:15.165 INFO:teuthology.orchestra.run.smithi172.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-22T06:15:15.215 INFO:teuthology.orchestra.run.smithi172.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-22T06:15:15.332 INFO:teuthology.orchestra.run.smithi172.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-22T06:15:15.458 INFO:teuthology.orchestra.run.smithi172.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-22T06:15:15.491 INFO:teuthology.orchestra.run.smithi172.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-04-22T06:15:15.550 INFO:teuthology.orchestra.run.smithi172.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-22T06:15:15.608 INFO:teuthology.orchestra.run.smithi172.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 6.0 MB/s | 358 kB 00:00 2024-04-22T06:15:15.650 INFO:teuthology.orchestra.run.smithi172.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-22T06:15:15.692 INFO:teuthology.orchestra.run.smithi172.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-04-22T06:15:15.734 INFO:teuthology.orchestra.run.smithi172.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2024-04-22T06:15:15.776 INFO:teuthology.orchestra.run.smithi172.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 469 kB/s | 19 kB 00:00 2024-04-22T06:15:15.818 INFO:teuthology.orchestra.run.smithi172.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-04-22T06:15:15.860 INFO:teuthology.orchestra.run.smithi172.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-04-22T06:15:15.902 INFO:teuthology.orchestra.run.smithi172.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-04-22T06:15:15.944 INFO:teuthology.orchestra.run.smithi172.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-04-22T06:15:15.986 INFO:teuthology.orchestra.run.smithi172.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-04-22T06:15:16.028 INFO:teuthology.orchestra.run.smithi172.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-04-22T06:15:16.070 INFO:teuthology.orchestra.run.smithi172.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 858 kB/s | 36 kB 00:00 2024-04-22T06:15:16.112 INFO:teuthology.orchestra.run.smithi172.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-04-22T06:15:16.162 INFO:teuthology.orchestra.run.smithi172.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-04-22T06:15:16.204 INFO:teuthology.orchestra.run.smithi172.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 481 kB/s | 20 kB 00:00 2024-04-22T06:15:17.080 INFO:teuthology.orchestra.run.smithi172.stdout:(35/36): ceph-mon-19.0.0-3189.g8d82defa.el9.x86 1.5 MB/s | 5.0 MB 00:03 2024-04-22T06:15:17.805 INFO:teuthology.orchestra.run.smithi172.stdout:(36/36): ceph-osd-19.0.0-3189.g8d82defa.el9.x86 18 MB/s | 71 MB 00:03 2024-04-22T06:15:17.808 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:15:17.808 INFO:teuthology.orchestra.run.smithi172.stdout:Total 18 MB/s | 84 MB 00:04 2024-04-22T06:15:17.947 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:15:17.991 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:15:17.991 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:15:18.603 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:15:18.603 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:15:20.171 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:15:20.240 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T06:15:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T06:15:20.318 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T06:15:20.390 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T06:15:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T06:15:20.466 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T06:15:20.558 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T06:15:20.637 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T06:15:20.686 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T06:15:20.729 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T06:15:20.759 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T06:15:20.794 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T06:15:20.827 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T06:15:20.868 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T06:15:20.911 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T06:15:20.946 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T06:15:20.984 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T06:15:21.057 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T06:15:21.195 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T06:15:21.283 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T06:15:21.345 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T06:15:21.383 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T06:15:21.432 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T06:15:21.488 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T06:15:21.673 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T06:15:21.721 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T06:15:21.799 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T06:15:21.891 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T06:15:21.937 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T06:15:22.054 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T06:15:22.237 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 31/36 2024-04-22T06:15:22.272 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:15:22.299 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 32/36 2024-04-22T06:15:22.299 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:15:22.299 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T06:15:22.299 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:15:22.299 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T06:15:22.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:33.149 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:15:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 33/36 2024-04-22T06:15:33.176 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:15:33.176 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T06:15:33.176 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:15:33.176 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T06:15:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:33.717 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:15:33.740 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 34/36 2024-04-22T06:15:33.741 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:15:33.741 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T06:15:33.741 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:15:33.741 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T06:15:33.742 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:15:33.999 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 35/36 2024-04-22T06:15:33.999 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:15:33.999 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T06:15:33.999 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:15:34.000 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T06:15:34.000 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:34.062 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:15:40.677 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 36/36 2024-04-22T06:15:40.677 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 1/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 3/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 4/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 5/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9. 6/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T06:15:40.678 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T06:15:40.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T06:15:40.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T06:15:40.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T06:15:40.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T06:15:40.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T06:15:40.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T06:15:40.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T06:15:40.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T06:15:40.681 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T06:15:41.770 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T06:15:41.770 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:41.770 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T06:15:41.771 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T06:15:41.772 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:41.773 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:42.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2024-04-22T06:15:42.658 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:37 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:42.737 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:15:42.779 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:15:42.780 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:42.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2024-04-22T06:15:43.351 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:43.470 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 762 k 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:43.471 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:15:43.472 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:43.472 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:15:43.472 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:43.472 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 762 k 2024-04-22T06:15:43.472 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 769 k 2024-04-22T06:15:43.472 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:15:43.916 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-19.0.0-3189.g8d82defa.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-22T06:15:43.917 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:15:43.917 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-22T06:15:43.917 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:15:43.922 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:15:43.922 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:15:43.928 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:15:43.928 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:15:43.996 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:15:44.372 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:15:44.461 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:15:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:15:45.039 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/1 2024-04-22T06:15:45.039 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:45.039 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:15:45.039 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:15:45.039 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:45.039 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:45.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T06:15:45.814 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:40 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:45.937 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:45.938 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-22T06:15:45.938 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:45.938 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:15:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 157 k 2024-04-22T06:15:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 157 k 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 640 k 2024-04-22T06:15:45.939 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:15:46.539 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-19.0.0-3189.g8d82de 262 kB/s | 157 kB 00:00 2024-04-22T06:15:46.540 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:15:46.540 INFO:teuthology.orchestra.run.smithi172.stdout:Total 261 kB/s | 157 kB 00:00 2024-04-22T06:15:46.540 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:15:46.550 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:15:46.550 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:15:46.606 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:15:46.607 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:15:46.758 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:15:46.849 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:15:46.873 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:15:46.873 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:15:46.873 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T06:15:46.873 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa. 1/1 2024-04-22T06:15:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:47.729 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:15:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:15:47.729 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:47.730 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:47.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2024-04-22T06:15:48.491 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:43 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:48.571 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:15:48.615 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:48.616 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:15:48.616 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:48.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T06:15:49.183 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:44 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:49.303 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 3.5 M 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 34 k 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 16 k 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T06:15:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout:Install 11 Packages 2024-04-22T06:15:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:49.307 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 7.6 M 2024-04-22T06:15:49.307 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 100 M 2024-04-22T06:15:49.307 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:15:50.086 INFO:teuthology.orchestra.run.smithi172.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3189.g8d8 54 kB/s | 16 kB 00:00 2024-04-22T06:15:50.111 INFO:teuthology.orchestra.run.smithi172.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3189.g8d 103 kB/s | 34 kB 00:00 2024-04-22T06:15:50.211 INFO:teuthology.orchestra.run.smithi172.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 8.4 MB/s | 863 kB 00:00 2024-04-22T06:15:50.528 INFO:teuthology.orchestra.run.smithi172.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-04-22T06:15:50.586 INFO:teuthology.orchestra.run.smithi172.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-04-22T06:15:50.779 INFO:teuthology.orchestra.run.smithi172.stdout:(6/11): ceph-mgr-dashboard-19.0.0-3189.g8d82def 3.5 MB/s | 3.5 MB 00:00 2024-04-22T06:15:50.795 INFO:teuthology.orchestra.run.smithi172.stdout:(7/11): python3-protobuf-3.14.0-13.el9.noarch.r 379 kB/s | 269 kB 00:00 2024-04-22T06:15:50.837 INFO:teuthology.orchestra.run.smithi172.stdout:(8/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.1 MB/s | 2.0 MB 00:00 2024-04-22T06:15:50.904 INFO:teuthology.orchestra.run.smithi172.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-04-22T06:15:50.946 INFO:teuthology.orchestra.run.smithi172.stdout:(10/11): python3-repoze-lru-0.7-16.el9.noarch.r 204 kB/s | 31 kB 00:00 2024-04-22T06:15:51.030 INFO:teuthology.orchestra.run.smithi172.stdout:(11/11): python3-grpcio-tools-1.46.7-10.el9.x86 576 kB/s | 144 kB 00:00 2024-04-22T06:15:51.031 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:15:51.032 INFO:teuthology.orchestra.run.smithi172.stdout:Total 4.4 MB/s | 7.6 MB 00:01 2024-04-22T06:15:51.062 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:15:51.079 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:15:51.079 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:15:51.227 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:15:51.227 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:15:51.710 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:15:51.795 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T06:15:51.825 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T06:15:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T06:15:52.027 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T06:15:52.078 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T06:15:52.163 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T06:15:52.205 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T06:15:52.234 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:15:52.266 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 9/11 2024-04-22T06:15:53.347 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 10/11 2024-04-22T06:15:53.374 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:15:54.461 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 11/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el 1/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noa 2/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9 3/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T06:15:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T06:15:54.744 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T06:15:54.744 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:54.744 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:15:54.744 INFO:teuthology.orchestra.run.smithi172.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T06:15:54.744 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:54.745 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:15:55.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T06:15:55.517 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:50 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:15:55.639 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================= 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================= 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 7.4 M 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T06:15:55.640 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================= 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout:Install 12 Packages 2024-04-22T06:15:55.641 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:15:55.642 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 38 M 2024-04-22T06:15:55.642 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 209 M 2024-04-22T06:15:55.642 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:15:56.405 INFO:teuthology.orchestra.run.smithi172.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 394 kB/s | 195 kB 00:00 2024-04-22T06:15:56.454 INFO:teuthology.orchestra.run.smithi172.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-04-22T06:15:56.688 INFO:teuthology.orchestra.run.smithi172.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-04-22T06:15:56.738 INFO:teuthology.orchestra.run.smithi172.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-04-22T06:15:56.789 INFO:teuthology.orchestra.run.smithi172.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-22T06:15:57.014 INFO:teuthology.orchestra.run.smithi172.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-04-22T06:15:57.089 INFO:teuthology.orchestra.run.smithi172.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-04-22T06:15:57.122 INFO:teuthology.orchestra.run.smithi172.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 661 kB/s | 803 kB 00:01 2024-04-22T06:15:57.206 INFO:teuthology.orchestra.run.smithi172.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-04-22T06:15:57.365 INFO:teuthology.orchestra.run.smithi172.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-3 5.1 MB/s | 7.4 MB 00:01 2024-04-22T06:15:57.507 INFO:teuthology.orchestra.run.smithi172.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-22T06:15:57.890 INFO:teuthology.orchestra.run.smithi172.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-22T06:15:57.891 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:15:57.891 INFO:teuthology.orchestra.run.smithi172.stdout:Total 17 MB/s | 38 MB 00:02 2024-04-22T06:15:58.241 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:15:58.269 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:15:58.269 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:15:58.682 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:15:58.683 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:15:59.517 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:15:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T06:15:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:15:59.723 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:15:59.764 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T06:15:59.944 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T06:15:59.980 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T06:16:00.057 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T06:16:00.702 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T06:16:00.820 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T06:16:02.058 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:16:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T06:16:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:16:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 12/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82d 1/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T06:16:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T06:16:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T06:16:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T06:16:05.921 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T06:16:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T06:16:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T06:16:05.922 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:05.922 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:06.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2024-04-22T06:16:06.835 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:01 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:06.956 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 49 k 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T06:16:06.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2024-04-22T06:16:06.959 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:06.960 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.2 M 2024-04-22T06:16:06.960 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2024-04-22T06:16:06.960 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:07.481 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): ceph-mgr-rook-19.0.0-3189.g8d82defa.el9 271 kB/s | 49 kB 00:00 2024-04-22T06:16:07.872 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 45 kB/s | 26 kB 00:00 2024-04-22T06:16:07.897 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 32 kB/s | 19 kB 00:00 2024-04-22T06:16:08.415 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 238 kB/s | 222 kB 00:00 2024-04-22T06:16:08.715 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 189 kB/s | 159 kB 00:00 2024-04-22T06:16:08.840 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 296 kB/s | 279 kB 00:00 2024-04-22T06:16:09.074 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 81 kB/s | 54 kB 00:00 2024-04-22T06:16:09.166 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 71 kB/s | 32 kB 00:00 2024-04-22T06:16:09.293 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 33 kB/s | 15 kB 00:00 2024-04-22T06:16:09.558 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 222 kB/s | 59 kB 00:00 2024-04-22T06:16:09.733 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-04-22T06:16:09.792 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 283 kB/s | 203 kB 00:00 2024-04-22T06:16:10.575 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 755 kB/s | 1.0 MB 00:01 2024-04-22T06:16:10.580 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:10.580 INFO:teuthology.orchestra.run.smithi172.stdout:Total 622 kB/s | 2.2 MB 00:03 2024-04-22T06:16:10.650 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:10.667 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:10.667 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:10.753 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:10.754 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:11.278 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:11.452 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T06:16:11.510 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T06:16:11.577 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T06:16:11.613 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T06:16:11.654 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T06:16:11.749 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T06:16:11.859 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T06:16:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T06:16:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T06:16:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T06:16:12.736 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T06:16:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T06:16:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 13/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 1/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T06:16:13.516 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T06:16:13.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T06:16:13.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T06:16:13.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T06:16:13.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T06:16:13.517 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T06:16:13.815 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T06:16:13.815 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:13.815 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:13.815 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T06:16:13.816 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T06:16:13.817 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T06:16:13.817 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T06:16:13.817 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T06:16:13.817 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:13.817 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:14.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T06:16:14.595 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:09 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:14.716 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 142 k 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:14.717 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:14.718 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:14.718 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-04-22T06:16:14.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:14.718 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 7.0 M 2024-04-22T06:16:14.718 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 33 M 2024-04-22T06:16:14.718 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:15.512 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 903 kB/s | 248 kB 00:00 2024-04-22T06:16:15.578 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3189.g8d82defa.e 416 kB/s | 142 kB 00:00 2024-04-22T06:16:15.671 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-04-22T06:16:15.696 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 492 kB/s | 58 kB 00:00 2024-04-22T06:16:15.721 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-04-22T06:16:15.796 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-22T06:16:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:15.797 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.5 MB/s | 7.0 MB 00:01 2024-04-22T06:16:15.868 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:15.879 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:15.879 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:15.976 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:15.977 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:16.279 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:16.443 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T06:16:16.497 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T06:16:17.009 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T06:16:17.091 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T06:16:17.155 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T06:16:17.185 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:16:17.947 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 6/6 2024-04-22T06:16:17.947 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 1/6 2024-04-22T06:16:17.947 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T06:16:17.947 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T06:16:17.947 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T06:16:17.947 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T06:16:18.198 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T06:16:18.198 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:18.199 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:18.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2024-04-22T06:16:18.987 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:13 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:19.108 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:19.108 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 871 k 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-04-22T06:16:19.109 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:19.110 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 951 k 2024-04-22T06:16:19.110 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 4.0 M 2024-04-22T06:16:19.110 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:19.473 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): ceph-fuse-19.0.0-3189.g8d82defa.el9.x86_ 2.9 MB/s | 871 kB 00:00 2024-04-22T06:16:19.615 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 184 kB/s | 80 kB 00:00 2024-04-22T06:16:19.615 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:19.615 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.8 MB/s | 951 kB 00:00 2024-04-22T06:16:19.627 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:19.650 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:19.651 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:19.727 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:19.728 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:19.907 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:20.042 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T06:16:20.104 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:16:20.772 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2/2 2024-04-22T06:16:20.772 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/2 2024-04-22T06:16:21.024 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T06:16:21.025 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:21.025 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:21.025 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T06:16:21.025 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:21.025 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:21.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2024-04-22T06:16:21.833 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:16 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:21.957 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:19.0.0-3189.g8d82defa.el9 ceph-noarch 273 k 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T06:16:21.958 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 350 k 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.5 M 2024-04-22T06:16:21.959 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:22.386 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 497 kB/s | 77 kB 00:00 2024-04-22T06:16:22.703 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-volume-19.0.0-3189.g8d82defa.el9.no 578 kB/s | 273 kB 00:00 2024-04-22T06:16:22.703 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:22.703 INFO:teuthology.orchestra.run.smithi172.stdout:Total 470 kB/s | 350 kB 00:00 2024-04-22T06:16:22.715 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:22.724 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:22.724 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:22.749 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:22.750 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:22.892 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:23.067 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T06:16:23.097 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:16:23.112 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2/2 2024-04-22T06:16:23.112 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:16:23.112 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T06:16:23.112 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:23.789 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 1/2 2024-04-22T06:16:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T06:16:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:24.072 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:19.0.0-3189.g8d82defa.el9.noarch 2024-04-22T06:16:24.072 INFO:teuthology.orchestra.run.smithi172.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T06:16:24.073 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:24.073 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:24.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2024-04-22T06:16:24.894 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:25.020 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 125 k 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:25.021 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:25.022 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:16:25.022 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:25.022 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 125 k 2024-04-22T06:16:25.022 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 445 k 2024-04-22T06:16:25.022 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:25.217 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-19.0.0-3189.g8d82defa.el9.x86_64 640 kB/s | 125 kB 00:00 2024-04-22T06:16:25.218 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:25.218 INFO:teuthology.orchestra.run.smithi172.stdout:Total 635 kB/s | 125 kB 00:00 2024-04-22T06:16:25.218 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:25.224 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:25.224 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:25.292 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:25.292 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:25.447 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:25.548 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:26.235 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:26.235 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:26.236 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:26.236 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:26.236 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:26.236 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:26.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2024-04-22T06:16:27.009 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:27.089 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:27.131 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:27.132 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:27.132 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:27.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2024-04-22T06:16:27.758 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:27.878 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 32 k 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:27.879 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:27.880 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:27.880 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:16:27.880 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:27.880 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 32 k 2024-04-22T06:16:27.880 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 142 k 2024-04-22T06:16:27.880 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:28.271 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-19.0.0-3189.g8d82defa.el9.x86_6 81 kB/s | 32 kB 00:00 2024-04-22T06:16:28.272 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:28.272 INFO:teuthology.orchestra.run.smithi172.stdout:Total 81 kB/s | 32 kB 00:00 2024-04-22T06:16:28.272 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:28.275 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:28.276 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:28.278 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:28.278 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:28.355 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:28.466 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:28.829 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:29.038 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:29.039 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:29.039 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:29.039 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:29.039 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:29.039 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:29.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2024-04-22T06:16:29.766 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:24 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:29.846 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:29.887 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:29.887 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:29.888 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:29.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2024-04-22T06:16:30.478 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:30.559 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:30.600 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:30.601 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:30.602 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:30.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2024-04-22T06:16:31.179 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:26 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:31.258 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:31.299 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:31.300 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:31.300 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:31.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2024-04-22T06:16:31.882 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:26 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:31.961 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:32.002 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:32.003 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:32.003 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:32.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2024-04-22T06:16:32.592 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:27 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:32.672 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:32.714 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:32.715 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:32.715 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:32.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2024-04-22T06:16:33.299 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:28 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:33.378 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:19.0.0-3189.g8d82defa.el9.x86_64 is already installed. 2024-04-22T06:16:33.420 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:33.421 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-04-22T06:16:33.421 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:33.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2024-04-22T06:16:34.008 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:29 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:34.128 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:34.129 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-22T06:16:34.129 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:34.129 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 90 k 2024-04-22T06:16:34.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 90 k 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 313 k 2024-04-22T06:16:34.130 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:34.374 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-19.0.0-3189.g8d82defa.el9.x86_64.rpm 369 kB/s | 90 kB 00:00 2024-04-22T06:16:34.375 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:34.375 INFO:teuthology.orchestra.run.smithi172.stdout:Total 366 kB/s | 90 kB 00:00 2024-04-22T06:16:34.375 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:34.384 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:34.384 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:34.446 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:34.446 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:34.590 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:34.678 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:35.057 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:35.320 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:35.320 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:35.320 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:35.320 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:35.320 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:35.321 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:35.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2024-04-22T06:16:36.119 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:31 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:36.240 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:36.240 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:36.240 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-04-22T06:16:36.240 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:36.240 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 3.4 M 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:36.241 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.4 M 2024-04-22T06:16:36.242 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 29 M 2024-04-22T06:16:36.242 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:36.619 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-19.0.0-3189.g8d82defa.el9.x86_64.rpm 9.0 MB/s | 3.4 MB 00:00 2024-04-22T06:16:36.620 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:36.620 INFO:teuthology.orchestra.run.smithi172.stdout:Total 9.0 MB/s | 3.4 MB 00:00 2024-04-22T06:16:36.620 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:36.631 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:36.631 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:36.720 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:36.720 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:37.353 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:37.378 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:37.378 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T06:16:37.378 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T06:16:37.379 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:16:37.379 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T06:16:37.379 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:38.589 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:38.590 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:38.590 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:38.590 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:38.590 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:38.590 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:38.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2024-04-22T06:16:39.345 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:34 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:16:39.470 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 175 k 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 175 k 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 673 k 2024-04-22T06:16:39.471 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:16:40.081 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-19.0.0-3189.g8d82defa.el9.x86_64.rpm 288 kB/s | 175 kB 00:00 2024-04-22T06:16:40.081 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:16:40.082 INFO:teuthology.orchestra.run.smithi172.stdout:Total 287 kB/s | 175 kB 00:00 2024-04-22T06:16:40.082 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:16:40.091 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:16:40.092 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:16:40.156 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:16:40.156 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:16:40.310 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:16:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:40.822 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 1/1 2024-04-22T06:16:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:41.046 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:16:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:16:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:41.046 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:16:41.318 DEBUG:teuthology.parallel:result is None 2024-04-22T06:16:41.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=8d82defa854b9197ed2c2b855ac5e996c480ec88 2024-04-22T06:16:41.485 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T06:16:41.508 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-3189.g8d82defa.el9 2024-04-22T06:16:41.508 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3189.g8d82defa.el9 2024-04-22T06:16:41.508 INFO:teuthology.task.install:The correct ceph version 19.0.0-3189.g8d82defa is installed. 2024-04-22T06:16:41.511 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T06:16:41.511 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:41.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T06:16:41.572 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T06:16:41.573 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:41.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T06:16:41.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T06:16:41.708 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T06:16:41.708 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:41.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T06:16:41.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T06:16:41.851 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T06:16:41.851 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:41.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T06:16:41.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T06:16:41.990 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T06:16:42.097 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T06:16:42.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-04-22T06:16:42.126 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T06:16:42.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T06:16:42.215 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T06:16:42.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T06:16:42.297 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T06:16:42.297 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T06:16:42.297 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184', 'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7668184', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-fix-tri-mutex-4-base', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'targets': {'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA0D68EFUFtLZRcE4tO2yC1Fv9tXMjfIvB1Rmx18Iz6fjurTGkPY1y6+HtXjK9pqji3SiR9CmYnGRGG68D1oNo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '8d82defa854b9197ed2c2b855ac5e996c480ec88', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-22_05:57:37', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T06:16:42.298 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T06:16:42.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T06:16:42.408 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:42.408 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T06:16:42.464 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T06:16:42.464 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-04-22T06:16:42.520 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 06:16:37.636489578 +0000 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 06:13:28.553056100 +0000 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 06:13:28.553056100 +0000 2024-04-22T06:16:42.521 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 06:13:28.553056100 +0000 2024-04-22T06:16:42.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T06:16:42.583 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T06:16:42.583 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T06:16:42.583 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000205736 s, 2.5 MB/s 2024-04-22T06:16:42.584 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T06:16:42.642 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 06:16:37.637489554 +0000 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 06:13:28.886047923 +0000 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 06:13:28.886047923 +0000 2024-04-22T06:16:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 06:13:28.886047923 +0000 2024-04-22T06:16:42.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T06:16:42.761 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T06:16:42.762 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T06:16:42.762 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.00019614 s, 2.6 MB/s 2024-04-22T06:16:42.763 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T06:16:42.823 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-04-22T06:16:42.879 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T06:16:42.879 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:16:42.879 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T06:16:42.879 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:16:42.879 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:16:42.879 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 06:16:37.637489554 +0000 2024-04-22T06:16:42.880 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 06:13:29.225039600 +0000 2024-04-22T06:16:42.880 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 06:13:29.225039600 +0000 2024-04-22T06:16:42.880 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 06:13:29.225039600 +0000 2024-04-22T06:16:42.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T06:16:42.944 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T06:16:42.944 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T06:16:42.944 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000206541 s, 2.5 MB/s 2024-04-22T06:16:42.945 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T06:16:43.006 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-04-22T06:16:43.065 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T06:16:43.065 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T06:16:43.065 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-22T06:16:43.065 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T06:16:43.065 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T06:16:43.065 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-04-22 06:16:37.637489554 +0000 2024-04-22T06:16:43.066 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-04-22 06:13:29.574031031 +0000 2024-04-22T06:16:43.066 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-04-22 06:13:29.574031031 +0000 2024-04-22T06:16:43.066 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-04-22 06:13:29.574031031 +0000 2024-04-22T06:16:43.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T06:16:43.130 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-04-22T06:16:43.130 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-04-22T06:16:43.130 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000207362 s, 2.5 MB/s 2024-04-22T06:16:43.132 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T06:16:43.190 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:16:43.191 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T06:16:43.191 INFO:tasks.ceph:Generating config... 2024-04-22T06:16:43.192 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-22T06:16:43.192 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T06:16:43.192 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-22T06:16:43.192 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-22T06:16:43.192 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T06:16:43.193 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-22T06:16:43.193 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-22T06:16:43.193 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-22T06:16:43.193 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T06:16:43.193 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T06:16:43.193 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] log to file = False 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T06:16:43.194 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] auth client required = none 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] auth service required = none 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] auth supported = none 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-22T06:16:43.195 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-22T06:16:43.196 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-22T06:16:43.197 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-22T06:16:43.198 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-22T06:16:43.199 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-22T06:16:43.200 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-22T06:16:43.200 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T06:16:43.200 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T06:16:43.200 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-22T06:16:43.200 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-22T06:16:43.200 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-22T06:16:43.200 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-22T06:16:43.200 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-22T06:16:43.200 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-22T06:16:43.200 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-22T06:16:43.200 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-22T06:16:43.201 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T06:16:43.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T06:16:43.285 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T06:16:43.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T06:16:43.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:16:43.363 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.172')] 2024-04-22T06:16:43.364 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-22T06:16:43.365 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:43.365 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T06:16:43.421 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:16:43.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:16:43.514+0000 7fa07a02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:43.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:16:43.514+0000 7fa07a02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:43.515 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:16:43.515+0000 7fa07a02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid 1e1e403f-0fda-4725-9a09-17e24f55cab1 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = quincy 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:fsid 1e1e403f-0fda-4725-9a09-17e24f55cab1 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2024-04-22T06:16:43.516169+0000 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:created 2024-04-22T06:16:43.516169+0000 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 17 (quincy) 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2024-04-22T06:16:43.516 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-22T06:16:43.518 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T06:16:43.538 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1e1e403f-0fda-4725-9a09-17e24f55cab1... 2024-04-22T06:16:43.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T06:16:43.622 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-04-22T06:16:43.622 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-04-22T06:16:43.622 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T06:16:43.622 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = none 2024-04-22T06:16:43.622 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.623 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-04-22T06:16:43.623 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.623 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-04-22T06:16:43.623 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.623 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-04-22T06:16:43.624 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-04-22T06:16:43.625 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: auth client required = none 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: auth cluster required = none 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: auth service required = none 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: debug asok = 0/0 2024-04-22T06:16:43.626 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug bdev = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug buffer = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug civetweb = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug client = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug compressor = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug context = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug crush = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug crypto = 0/0 2024-04-22T06:16:43.627 INFO:teuthology.orchestra.run.smithi172.stdout: debug dpdk = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug eventtrace = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug filer = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug filestore = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug finisher = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug fuse = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug heartbeatmap = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug javaclient = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug journal = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug journaler = 0/0 2024-04-22T06:16:43.628 INFO:teuthology.orchestra.run.smithi172.stdout: debug kstore = 0/0 2024-04-22T06:16:43.630 INFO:teuthology.orchestra.run.smithi172.stdout: debug leveldb = 0/0 2024-04-22T06:16:43.630 INFO:teuthology.orchestra.run.smithi172.stdout: debug lockdep = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds balancer = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds locker = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds log = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds log expire = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mds migrator = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug memdb = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 0/0 2024-04-22T06:16:43.631 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgrc = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug objclass = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug objectcacher = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug objecter = 0/0 2024-04-22T06:16:43.632 INFO:teuthology.orchestra.run.smithi172.stdout: debug optracker = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug perfcounter = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug rados = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd mirror = 0/0 2024-04-22T06:16:43.633 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd replay = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug refs = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw lifecycle = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw notification = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug rgw sync = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug striper = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug throttle = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug timer = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: debug tp = 0/0 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: ms client mode = crc 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: ms cluster mode = crc 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon client mode = crc 2024-04-22T06:16:43.634 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon cluster mode = crc 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: ms mon service mode = crc 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: ms service mode = crc 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 1e1e403f-0fda-4725-9a09-17e24f55cab1 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.635 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-04-22T06:16:43.636 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression algorithm = snappy 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression mode = aggressive 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore rocksdb cf = False 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: crimson osd obc lru size = 10 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug filestore = 0/0 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug journal = 0/0 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 0/0 2024-04-22T06:16:43.637 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 0/0 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: err to stderr = True 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: log flush on exit = True 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: log to file = False 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: log to stderr = True 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-04-22T06:16:43.638 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 0/0 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 0/0 2024-04-22T06:16:43.639 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 0/0 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T06:16:43.640 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default crimson = True 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-04-22T06:16:43.641 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 0-2 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-04-22T06:16:43.642 INFO:teuthology.orchestra.run.smithi172.stdout: crimson seastar cpu cores = 3-5 2024-04-22T06:16:43.643 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T06:16:43.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T06:16:43.750 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T06:16:43.750 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:43.750 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T06:16:43.768 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:43.768 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T06:16:43.828 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:16:43.829 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:43.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T06:16:43.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T06:16:43.904 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:16:43.904 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:16:43.961 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T06:16:43.961 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T06:16:43.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T06:16:44.059 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T06:16:44.062 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T06:16:44.062 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T06:16:44.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T06:16:44.111 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T06:16:44.124 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T06:16:44.124 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T06:16:44.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T06:16:44.151 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:16:44.151 INFO:tasks.ceph:role: osd.0 2024-04-22T06:16:44.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:16:44.152 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T06:16:44.224 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:16:44.225 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:16:44.741 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-04-22T06:16:44.743 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-04-22T06:16:44.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T06:16:44.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T06:16:44.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T06:16:44.958 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:16:44.959 INFO:tasks.ceph:role: osd.1 2024-04-22T06:16:44.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:16:44.959 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T06:16:45.031 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:16:45.031 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:16:45.031 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:16:45.031 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:16:45.031 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:16:45.031 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:16:45.032 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:16:45.032 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:16:45.032 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:16:45.032 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:16:45.490 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-04-22T06:16:45.492 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-04-22T06:16:45.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T06:16:45.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T06:16:45.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T06:16:45.673 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T06:16:45.674 INFO:tasks.ceph:role: osd.2 2024-04-22T06:16:45.674 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:16:45.674 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T06:16:45.746 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T06:16:46.236 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-04-22T06:16:46.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-04-22T06:16:46.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T06:16:46.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T06:16:46.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:16:46.690 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:16:46.690 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-22 06:16:46,690 seastar - Reactor backend: linux-aio 2024-04-22T06:16:46.699 INFO:teuthology.orchestra.run.smithi172.stderr:==26875==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:16:46.819 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-04-22 06:16:46,820 [shard 0:main] seastar - reactor::drain 2024-04-22T06:16:47.345 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:47.346 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-22T06:16:47.346 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-22T06:16:47.346 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-22T06:16:47.346 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:47.346 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:16:47.402 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:16:47.406 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-22 06:16:47,407 seastar - Reactor backend: linux-aio 2024-04-22T06:16:47.414 INFO:teuthology.orchestra.run.smithi172.stderr:==26873==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:16:47.480 INFO:teuthology.orchestra.run.smithi172.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x113bc7 0x4a0bd 0x35b7c 0x3689e 0x5db8f5f 0x5db912a 0x5db96b6 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x368158a 0x2958f 0x2963f 0x346d444 2024-04-22T06:16:47.480 INFO:teuthology.orchestra.run.smithi172.stderr:kernel callstack: 2024-04-22T06:16:52.047 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:52.048 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-22T06:16:52.048 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-22T06:16:52.048 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-22T06:16:52.048 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:52.048 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:16:52.048 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 1e1e403f-0fda-4725-9a09-17e24f55cab1 2024-04-22T06:16:52.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T06:16:52.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:16:52.377 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:16:52.378 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-22 06:16:52,379 seastar - Reactor backend: linux-aio 2024-04-22T06:16:52.382 INFO:teuthology.orchestra.run.smithi172.stderr:==27057==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:16:52.485 INFO:teuthology.orchestra.run.smithi172.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9e61c9 0xb8b8c90 0xb8ba20c 0xb8ba422 0xb8ba578 0xb8baa41 0x3e6ef 0x5e3a256 0x5e3a469 0x5db8f50 0x5db912a 0x5db96b6 0x5bfe1ef 0x5c192b3 0x6ef1c24 0x6f033f7 0x6f04894 0x6f04919 0x6f04a1b 0x6f04b13 0x6f04d20 0x6f04f06 0x6f05255 0x6f055f9 0x6f05829 0x6f05a93 0x6f05d60 0x6f05f7e 0x6f06036 0x6f060fc 0x6f061dd 0x5a66ae8 0x5a66bf2 0x5a66cae 0x5a66d6a 0x5a66e26 0x5a66f73 0x5a670fe 0x5a674bf 0x5a83400 0xb8cb915 0xb8e5baf 0xb986bad 0xb9881fb 0xb6649e1 0xb66535d 0x37e091a 0x37e1473 0x3680d7a 0x2958f 0x2963f 0x346d444 2024-04-22T06:16:52.485 INFO:teuthology.orchestra.run.smithi172.stderr:kernel callstack: 2024-04-22T06:16:52.502 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-04-22 06:16:52,503 [shard 0:main] seastar - reactor::drain 2024-04-22T06:16:53.036 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:53.036 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-22T06:16:53.036 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-22T06:16:53.036 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-22T06:16:53.036 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:53.036 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:16:53.090 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:16:53.094 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-22 06:16:53,095 seastar - Reactor backend: linux-aio 2024-04-22T06:16:53.110 INFO:teuthology.orchestra.run.smithi172.stderr:==27055==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:16:57.752 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:57.753 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-22T06:16:57.753 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-22T06:16:57.753 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-22T06:16:57.753 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:57.753 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:16:57.754 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 1e1e403f-0fda-4725-9a09-17e24f55cab1 2024-04-22T06:16:57.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T06:16:57.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:16:58.079 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:16:58.080 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-22 06:16:58,081 seastar - Reactor backend: linux-aio 2024-04-22T06:16:58.085 INFO:teuthology.orchestra.run.smithi172.stderr:==27243==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:16:58.205 INFO:teuthology.orchestra.run.smithi172.stderr:DEBUG 2024-04-22 06:16:58,206 [shard 0:main] seastar - reactor::drain 2024-04-22T06:16:58.740 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:58.740 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-22T06:16:58.740 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-22T06:16:58.740 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-22T06:16:58.740 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:16:58.740 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:16:58.797 INFO:teuthology.orchestra.run.smithi172.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T06:16:58.801 INFO:teuthology.orchestra.run.smithi172.stderr:INFO 2024-04-22 06:16:58,802 seastar - Reactor backend: linux-aio 2024-04-22T06:16:58.813 INFO:teuthology.orchestra.run.smithi172.stderr:==27241==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stderr:Suppressions used: 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stderr: count bytes template 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stderr: 1 8 MallocExtension::Register 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stderr:----------------------------------------------------- 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:17:03.474 INFO:teuthology.orchestra.run.smithi172.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 1e1e403f-0fda-4725-9a09-17e24f55cab1 2024-04-22T06:17:03.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T06:17:03.521 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T06:17:03.521 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:03.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T06:17:03.585 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:03.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T06:17:03.648 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:03.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T06:17:03.712 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:03.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T06:17:03.775 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:03.775 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T06:17:03.830 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T06:17:03.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T06:17:03.892 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-04-22T06:17:03.892 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDMACZms32ZAxAAS93qqjwhcH+9s0jVimb4ug== 2024-04-22T06:17:03.892 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDPACZm76LnIBAAKBRE675ZpMhJBpmZOtdNww== 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDVACZmt96eDhAAXGO7W0oBsknhrRiBQQMp5A== 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDaACZmXkvAOBAAg6krIha5SIpdgo2Jwqd8GA== 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-04-22T06:17:03.893 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDMACZmLcGzBhAAYaSl4wWN3i1Cng/aZByi2A== 2024-04-22T06:17:03.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T06:17:03.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:17:04.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:17:04.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T06:17:04.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T06:17:04.231 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T06:17:04.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T06:17:04.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T06:17:04.363 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:04.363+0000 7fb3ff08ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.363 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:04.363+0000 7fb3ff08ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T06:17:04.400 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T06:17:04.455 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T06:17:04.456 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T06:17:04.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T06:17:04.498 INFO:tasks.ceph.mon.a:Started 2024-04-22T06:17:04.499 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T06:17:04.499 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T06:17:04.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T06:17:04.501 INFO:tasks.ceph.mgr.x:Started 2024-04-22T06:17:04.501 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T06:17:04.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T06:17:04.580 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-04-22T06:17:04.580+0000 7f274645ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.580 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-04-22T06:17:04.580+0000 7f274645ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.580 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-04-22T06:17:04.580+0000 7f274645ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.582 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:04.582+0000 7f4058c2d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.611 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:04.606+0000 7f7ef046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.613 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:04.613+0000 7f7ef046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.682 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2024-04-22T06:17:04.693 INFO:tasks.ceph:check_enable_crimson: True 2024-04-22T06:17:04.693 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-22T06:17:04.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-22T06:17:04.785 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:04.785+0000 7f4058c2d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.786 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:04.786+0000 7f4058c2d000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.797 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:04.793+0000 7f402c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.799 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:04.799+0000 7f402c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:04.882 INFO:tasks.ceph.mon.a.smithi172.stderr:2024-04-22T06:17:04.882+0000 7f273d1f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-22T06:17:04.882 INFO:tasks.ceph.mon.a.smithi172.stderr:Please be aware that this feature is experimental, untested, 2024-04-22T06:17:04.882 INFO:tasks.ceph.mon.a.smithi172.stderr:unsupported, and may result in data corruption, data loss, 2024-04-22T06:17:04.882 INFO:tasks.ceph.mon.a.smithi172.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-22T06:17:04.882 INFO:tasks.ceph.mon.a.smithi172.stderr:feature with important data. 2024-04-22T06:17:04.882 INFO:tasks.ceph.mon.a.smithi172.stderr: 2024-04-22T06:17:04.891 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:04.891+0000 7f4058c2d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T06:17:05.038 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:05.038+0000 7f4058c2d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:17:05.092 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:05.092+0000 7f4058c2d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T06:17:05.348 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:05.348+0000 7f4058c2d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T06:17:05.394 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:05.394+0000 7f4058c2d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T06:17:05.694 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T06:17:05.694 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:05.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T06:17:05.717 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:05.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T06:17:05.781 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:05.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T06:17:05.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 932e7978-17ef-41a6-8678-992bd046c4f4 0 2024-04-22T06:17:05.993 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:05.988+0000 7f894f2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:05.995 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:05.995+0000 7f894f2ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.132 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:06.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 5e4311da-2991-4f70-8d36-230e9141c659 1 2024-04-22T06:17:06.209 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.209+0000 7f4058c2d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T06:17:06.252 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:06.245+0000 7f501886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.254 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:06.254+0000 7f501886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.306 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.306+0000 7f4058c2d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T06:17:06.338 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-04-22T06:17:06.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 6f5c8ea0-740a-4ca9-9da0-ff29672ecdbd 2 2024-04-22T06:17:06.352 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.351+0000 7f4058c2d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T06:17:06.456 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:06.450+0000 7f9677ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.458 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:06.458+0000 7f9677ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.527 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.527+0000 7f4058c2d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T06:17:06.534 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-04-22T06:17:06.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-22T06:17:06.574 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T06:17:06.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T06:17:06.579 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.579+0000 7f4058c2d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T06:17:06.581 INFO:tasks.ceph.osd.0:Started 2024-04-22T06:17:06.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-22T06:17:06.609 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T06:17:06.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T06:17:06.638 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.638+0000 7f4058c2d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T06:17:06.640 INFO:tasks.ceph.osd.1:Started 2024-04-22T06:17:06.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-22T06:17:06.668 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T06:17:06.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T06:17:06.711 INFO:tasks.ceph.osd.2:Started 2024-04-22T06:17:06.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:17:06.774 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:06.774+0000 7f4058c2d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T06:17:06.893 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:06.888+0000 7f7163510640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.896 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:06.896+0000 7f7163510640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:06.968 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:06.968 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":6,"fsid":"1e1e403f-0fda-4725-9a09-17e24f55cab1","created":"2024-04-22T06:17:04.601451+0000","modified":"2024-04-22T06:17:06.534860+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T06:17:06.534860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"932e7978-17ef-41a6-8678-992bd046c4f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5e4311da-2991-4f70-8d36-230e9141c659","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f5c8ea0-740a-4ca9-9da0-ff29672ecdbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:17:06.982 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T06:17:06.983 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T06:17:07.231 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:07.231+0000 7f4058c2d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T06:17:07.285 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T06:17:07.387 INFO:teuthology.misc.health.smithi172.stderr:2024-04-22T06:17:07.383+0000 7fc877d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:07.389 INFO:teuthology.misc.health.smithi172.stderr:2024-04-22T06:17:07.389+0000 7fc877d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:07.457 INFO:teuthology.misc.health.smithi172.stdout: 2024-04-22T06:17:07.457 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":6,"fsid":"1e1e403f-0fda-4725-9a09-17e24f55cab1","created":"2024-04-22T06:17:04.601451+0000","modified":"2024-04-22T06:17:06.534860+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T06:17:06.534860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"932e7978-17ef-41a6-8678-992bd046c4f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5e4311da-2991-4f70-8d36-230e9141c659","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f5c8ea0-740a-4ca9-9da0-ff29672ecdbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:17:07.468 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-22T06:17:07.523 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:07.520+0000 7f4058c2d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T06:17:07.579 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:07.579+0000 7f4058c2d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T06:17:07.711 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:07.710+0000 7f4058c2d000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T06:17:08.015 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.015+0000 7f4058c2d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T06:17:08.069 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.068+0000 7f4058c2d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T06:17:08.241 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.241+0000 7f4058c2d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T06:17:08.377 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.377+0000 7f4058c2d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T06:17:08.429 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.429+0000 7f4058c2d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T06:17:08.503 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.503+0000 7f4058c2d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T06:17:08.552 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:08.552+0000 7f4058c2d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T06:17:09.061 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:09.061+0000 7f4058c2d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T06:17:09.202 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:09.202+0000 7f4058c2d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T06:17:10.212 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-04-22T06:17:10.212+0000 7f3ff78fc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-22T06:17:13.771 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T06:17:13.884 INFO:teuthology.misc.health.smithi172.stderr:2024-04-22T06:17:13.880+0000 7f1ddbf54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:13.886 INFO:teuthology.misc.health.smithi172.stderr:2024-04-22T06:17:13.886+0000 7f1ddbf54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:14.073 INFO:teuthology.misc.health.smithi172.stdout: 2024-04-22T06:17:14.073 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":9,"fsid":"1e1e403f-0fda-4725-9a09-17e24f55cab1","created":"2024-04-22T06:17:04.601451+0000","modified":"2024-04-22T06:17:13.217620+0000","last_up_change":"2024-04-22T06:17:13.217620+0000","last_in_change":"2024-04-22T06:17:06.534860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"932e7978-17ef-41a6-8678-992bd046c4f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":391749019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":391749019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":391749019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":391749019}]},"public_addr":"172.21.15.172:6808/391749019","cluster_addr":"172.21.15.172:6809/391749019","heartbeat_back_addr":"172.21.15.172:6813/391749019","heartbeat_front_addr":"172.21.15.172:6812/391749019","state":["exists","up"]},{"osd":1,"uuid":"5e4311da-2991-4f70-8d36-230e9141c659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":3917744907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3917744907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3917744907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":3917744907}]},"public_addr":"172.21.15.172:6805/3917744907","cluster_addr":"172.21.15.172:6804/3917744907","heartbeat_back_addr":"172.21.15.172:6806/3917744907","heartbeat_front_addr":"172.21.15.172:6807/3917744907","state":["exists","up"]},{"osd":2,"uuid":"6f5c8ea0-740a-4ca9-9da0-ff29672ecdbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":4160750256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":4160750256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":4160750256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":4160750256}]},"public_addr":"172.21.15.172:6810/4160750256","cluster_addr":"172.21.15.172:6811/4160750256","heartbeat_back_addr":"172.21.15.172:6814/4160750256","heartbeat_front_addr":"172.21.15.172:6803/4160750256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:17:14.084 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-22T06:17:14.085 INFO:tasks.ceph:Creating RBD pool 2024-04-22T06:17:14.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T06:17:14.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:14.186+0000 7f6d36b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:14.194 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:14.194+0000 7f6d36b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:15.228 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2024-04-22T06:17:15.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T06:17:15.374 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:15.365+0000 7fcb41c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:15.376 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:15.376+0000 7fcb41c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:16.229 INFO:teuthology.orchestra.run.smithi172.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T06:17:16.250 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T06:17:16.251 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T06:17:16.251 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T06:17:16.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T06:17:16.252 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T06:17:16.408 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:16.405+0000 7f9f7a020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:16.411 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:16.411+0000 7f9f7a020640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:16.671 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:16.685 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3076149545},{"type":"v1","addr":"172.21.15.172:6801","nonce":3076149545}]},"active_addr":"172.21.15.172:6801/3076149545","active_change":"2024-04-22T06:17:09.204815+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1855449590}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1775400895}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":132302711}]}]} 2024-04-22T06:17:16.689 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T06:17:16.689 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T06:17:16.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:17:16.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:16.834+0000 7f12b4b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:16.841 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:16.841+0000 7f12b4b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.123 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:17.123 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":11,"fsid":"1e1e403f-0fda-4725-9a09-17e24f55cab1","created":"2024-04-22T06:17:04.601451+0000","modified":"2024-04-22T06:17:16.229398+0000","last_up_change":"2024-04-22T06:17:13.217620+0000","last_in_change":"2024-04-22T06:17:06.534860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T06:17:14.386476+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-22T06:17:15.224480+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"932e7978-17ef-41a6-8678-992bd046c4f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":391749019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":391749019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":391749019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":391749019}]},"public_addr":"172.21.15.172:6808/391749019","cluster_addr":"172.21.15.172:6809/391749019","heartbeat_back_addr":"172.21.15.172:6813/391749019","heartbeat_front_addr":"172.21.15.172:6812/391749019","state":["exists","up"]},{"osd":1,"uuid":"5e4311da-2991-4f70-8d36-230e9141c659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":3917744907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3917744907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3917744907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":3917744907}]},"public_addr":"172.21.15.172:6805/3917744907","cluster_addr":"172.21.15.172:6804/3917744907","heartbeat_back_addr":"172.21.15.172:6806/3917744907","heartbeat_front_addr":"172.21.15.172:6807/3917744907","state":["exists","up"]},{"osd":2,"uuid":"6f5c8ea0-740a-4ca9-9da0-ff29672ecdbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":4160750256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":4160750256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":4160750256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":4160750256}]},"public_addr":"172.21.15.172:6810/4160750256","cluster_addr":"172.21.15.172:6811/4160750256","heartbeat_back_addr":"172.21.15.172:6814/4160750256","heartbeat_front_addr":"172.21.15.172:6803/4160750256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:17:17.138 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T06:17:17.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T06:17:17.261 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.257+0000 7f9b7bd1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.264 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.264+0000 7f9b7bd1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.463 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:17.463 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":12,"fsid":"1e1e403f-0fda-4725-9a09-17e24f55cab1","created":"2024-04-22T06:17:04.601451+0000","modified":"2024-04-22T06:17:17.233887+0000","last_up_change":"2024-04-22T06:17:13.217620+0000","last_in_change":"2024-04-22T06:17:06.534860+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-22T06:17:14.386476+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-22T06:17:15.224480+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"932e7978-17ef-41a6-8678-992bd046c4f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":391749019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":391749019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":391749019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":391749019}]},"public_addr":"172.21.15.172:6808/391749019","cluster_addr":"172.21.15.172:6809/391749019","heartbeat_back_addr":"172.21.15.172:6813/391749019","heartbeat_front_addr":"172.21.15.172:6812/391749019","state":["exists","up"]},{"osd":1,"uuid":"5e4311da-2991-4f70-8d36-230e9141c659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6805","nonce":3917744907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3917744907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3917744907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6807","nonce":3917744907}]},"public_addr":"172.21.15.172:6805/3917744907","cluster_addr":"172.21.15.172:6804/3917744907","heartbeat_back_addr":"172.21.15.172:6806/3917744907","heartbeat_front_addr":"172.21.15.172:6807/3917744907","state":["exists","up"]},{"osd":2,"uuid":"6f5c8ea0-740a-4ca9-9da0-ff29672ecdbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":4160750256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6811","nonce":4160750256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":4160750256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6803","nonce":4160750256}]},"public_addr":"172.21.15.172:6810/4160750256","cluster_addr":"172.21.15.172:6811/4160750256","heartbeat_back_addr":"172.21.15.172:6814/4160750256","heartbeat_front_addr":"172.21.15.172:6803/4160750256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T06:17:17.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T06:17:17.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T06:17:17.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T06:17:17.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.588+0000 7f140d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.592 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.588+0000 7fec3fced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.593 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.588+0000 7f076326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.594 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.594+0000 7f140d19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.594+0000 7fec3fced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.595 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.595+0000 7f076326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.625 INFO:teuthology.orchestra.run.smithi172.stdout:38654705668 2024-04-22T06:17:17.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:17:17.627 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2024-04-22T06:17:17.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:17:17.628 INFO:teuthology.orchestra.run.smithi172.stdout:38654705668 2024-04-22T06:17:17.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:17:17.780 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.776+0000 7fec1c82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.776+0000 7fb9a9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.781 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.776+0000 7f08cfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.783+0000 7fb9a9c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.783+0000 7fec1c82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.783 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:17.783+0000 7f08cfc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:17.976 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:17.984 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:17.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:17:17.990 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:17.999 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.0 2024-04-22T06:17:18.004 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T06:17:18.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:17:19.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:17:19.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:17:19.105 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:19.100+0000 7fbb1475d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.108 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:19.108+0000 7fbb1475d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.151 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:19.147+0000 7fd62515d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.153 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:19.153+0000 7fd62515d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.159 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:19.154+0000 7f91d4b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.162 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:19.162+0000 7f91d4b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:19.355 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:19.360 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:19.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.0 2024-04-22T06:17:19.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:17:19.386 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:19.399 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T06:17:20.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:17:20.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:17:20.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:17:20.490 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:20.483+0000 7fb9afb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.493 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:20.493+0000 7fb9afb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:20.523+0000 7f2b924ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.528 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:20.523+0000 7fce440ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.530 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:20.530+0000 7f2b924ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.531 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:20.531+0000 7fce440ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:20.707 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:20.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.0 2024-04-22T06:17:20.725 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:20.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:17:20.751 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:20.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T06:17:21.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:17:21.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:17:21.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:17:21.845 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:21.837+0000 7fa6838ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:21.848 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:21.848+0000 7fa6838ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:21.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:21.876+0000 7f5ea386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:21.880 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:21.876+0000 7f6357a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:21.882 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:21.882+0000 7f5ea386d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:21.883 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:21.882+0000 7f6357a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:22.046 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:22.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.0 2024-04-22T06:17:22.081 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:22.087 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-04-22T06:17:22.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T06:17:22.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T06:17:23.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T06:17:23.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T06:17:23.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T06:17:23.190 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.182+0000 7f28f302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.192 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.193+0000 7f28f1dcb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.226 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.221+0000 7fcbf6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.221+0000 7fa93b59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.229 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.229+0000 7fcbf6ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.231 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.231+0000 7fa93b59d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.399 INFO:teuthology.orchestra.run.smithi172.stdout:38654705668 2024-04-22T06:17:23.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-04-22T06:17:23.414 DEBUG:teuthology.parallel:result is None 2024-04-22T06:17:23.435 INFO:teuthology.orchestra.run.smithi172.stdout:38654705669 2024-04-22T06:17:23.446 INFO:teuthology.orchestra.run.smithi172.stdout:34359738377 2024-04-22T06:17:23.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705669 for osd.2 2024-04-22T06:17:23.448 DEBUG:teuthology.parallel:result is None 2024-04-22T06:17:23.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738377 for osd.1 2024-04-22T06:17:23.459 DEBUG:teuthology.parallel:result is None 2024-04-22T06:17:23.459 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T06:17:23.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:17:23.607 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.604+0000 7f458082d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.609 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.609+0000 7f458082d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.791 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:23.791 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-04-22T06:17:23.804 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-22T06:17:23.210131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":421888,"data_stored":1249884,"data_compressed":59940,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.966195"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.272981+0000","last_change":"2024-04-22T06:17:16.340534+0000","last_active":"2024-04-22T06:17:18.272981+0000","last_peered":"2024-04-22T06:17:18.272981+0000","last_clean":"2024-04-22T06:17:18.272981+0000","last_became_active":"2024-04-22T06:17:16.338065+0000","last_became_peered":"2024-04-22T06:17:16.338065+0000","last_unstale":"2024-04-22T06:17:18.272981+0000","last_undegraded":"2024-04-22T06:17:18.272981+0000","last_fullsized":"2024-04-22T06:17:18.272981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.270667+0000","last_change":"2024-04-22T06:17:16.346150+0000","last_active":"2024-04-22T06:17:18.270667+0000","last_peered":"2024-04-22T06:17:18.270667+0000","last_clean":"2024-04-22T06:17:18.270667+0000","last_became_active":"2024-04-22T06:17:16.339087+0000","last_became_peered":"2024-04-22T06:17:16.339087+0000","last_unstale":"2024-04-22T06:17:18.270667+0000","last_undegraded":"2024-04-22T06:17:18.270667+0000","last_fullsized":"2024-04-22T06:17:18.270667+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:17.266268+0000","last_change":"2024-04-22T06:17:16.379289+0000","last_active":"2024-04-22T06:17:17.266268+0000","last_peered":"2024-04-22T06:17:17.266268+0000","last_clean":"2024-04-22T06:17:17.266268+0000","last_became_active":"2024-04-22T06:17:16.374817+0000","last_became_peered":"2024-04-22T06:17:16.374817+0000","last_unstale":"2024-04-22T06:17:17.266268+0000","last_undegraded":"2024-04-22T06:17:17.266268+0000","last_fullsized":"2024-04-22T06:17:17.266268+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.272095+0000","last_change":"2024-04-22T06:17:16.342100+0000","last_active":"2024-04-22T06:17:18.272095+0000","last_peered":"2024-04-22T06:17:18.272095+0000","last_clean":"2024-04-22T06:17:18.272095+0000","last_became_active":"2024-04-22T06:17:16.339266+0000","last_became_peered":"2024-04-22T06:17:16.339266+0000","last_unstale":"2024-04-22T06:17:18.272095+0000","last_undegraded":"2024-04-22T06:17:18.272095+0000","last_fullsized":"2024-04-22T06:17:18.272095+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'96","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:17.378906+0000","last_change":"2024-04-22T06:17:16.310856+0000","last_active":"2024-04-22T06:17:17.378906+0000","last_peered":"2024-04-22T06:17:17.378906+0000","last_clean":"2024-04-22T06:17:17.378906+0000","last_became_active":"2024-04-22T06:17:16.306067+0000","last_became_peered":"2024-04-22T06:17:16.306067+0000","last_unstale":"2024-04-22T06:17:17.378906+0000","last_undegraded":"2024-04-22T06:17:17.378906+0000","last_fullsized":"2024-04-22T06:17:17.378906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.272616+0000","last_change":"2024-04-22T06:17:16.319392+0000","last_active":"2024-04-22T06:17:18.272616+0000","last_peered":"2024-04-22T06:17:18.272616+0000","last_clean":"2024-04-22T06:17:18.272616+0000","last_became_active":"2024-04-22T06:17:16.316499+0000","last_became_peered":"2024-04-22T06:17:16.316499+0000","last_unstale":"2024-04-22T06:17:18.272616+0000","last_undegraded":"2024-04-22T06:17:18.272616+0000","last_fullsized":"2024-04-22T06:17:18.272616+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T06:17:16.308561+0000","last_change":"2024-04-22T06:17:16.309062+0000","last_active":"2024-04-22T06:17:16.308561+0000","last_peered":"2024-04-22T06:17:16.308561+0000","last_clean":"2024-04-22T06:17:15.226434+0000","last_became_active":"2024-04-22T06:17:16.297057+0000","last_became_peered":"2024-04-22T06:17:16.297057+0000","last_unstale":"2024-04-22T06:17:16.308561+0000","last_undegraded":"2024-04-22T06:17:16.308561+0000","last_fullsized":"2024-04-22T06:17:16.308561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:17.264188+0000","last_change":"2024-04-22T06:17:16.346365+0000","last_active":"2024-04-22T06:17:17.264188+0000","last_peered":"2024-04-22T06:17:17.264188+0000","last_clean":"2024-04-22T06:17:17.264188+0000","last_became_active":"2024-04-22T06:17:16.341358+0000","last_became_peered":"2024-04-22T06:17:16.341358+0000","last_unstale":"2024-04-22T06:17:17.264188+0000","last_undegraded":"2024-04-22T06:17:17.264188+0000","last_fullsized":"2024-04-22T06:17:17.264188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.269969+0000","last_change":"2024-04-22T06:17:16.360340+0000","last_active":"2024-04-22T06:17:18.269969+0000","last_peered":"2024-04-22T06:17:18.269969+0000","last_clean":"2024-04-22T06:17:18.269969+0000","last_became_active":"2024-04-22T06:17:16.355766+0000","last_became_peered":"2024-04-22T06:17:16.355766+0000","last_unstale":"2024-04-22T06:17:18.269969+0000","last_undegraded":"2024-04-22T06:17:18.269969+0000","last_fullsized":"2024-04-22T06:17:18.269969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614505,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614505,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":20874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:17:23.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T06:17:23.911 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.908+0000 7f373222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:23.913 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:23.913+0000 7f373222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:24.098 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:24.098 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-04-22T06:17:24.110 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-22T06:17:23.210131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":421888,"data_stored":1249884,"data_compressed":59940,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.966195"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.272981+0000","last_change":"2024-04-22T06:17:16.340534+0000","last_active":"2024-04-22T06:17:18.272981+0000","last_peered":"2024-04-22T06:17:18.272981+0000","last_clean":"2024-04-22T06:17:18.272981+0000","last_became_active":"2024-04-22T06:17:16.338065+0000","last_became_peered":"2024-04-22T06:17:16.338065+0000","last_unstale":"2024-04-22T06:17:18.272981+0000","last_undegraded":"2024-04-22T06:17:18.272981+0000","last_fullsized":"2024-04-22T06:17:18.272981+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.270667+0000","last_change":"2024-04-22T06:17:16.346150+0000","last_active":"2024-04-22T06:17:18.270667+0000","last_peered":"2024-04-22T06:17:18.270667+0000","last_clean":"2024-04-22T06:17:18.270667+0000","last_became_active":"2024-04-22T06:17:16.339087+0000","last_became_peered":"2024-04-22T06:17:16.339087+0000","last_unstale":"2024-04-22T06:17:18.270667+0000","last_undegraded":"2024-04-22T06:17:18.270667+0000","last_fullsized":"2024-04-22T06:17:18.270667+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:17.266268+0000","last_change":"2024-04-22T06:17:16.379289+0000","last_active":"2024-04-22T06:17:17.266268+0000","last_peered":"2024-04-22T06:17:17.266268+0000","last_clean":"2024-04-22T06:17:17.266268+0000","last_became_active":"2024-04-22T06:17:16.374817+0000","last_became_peered":"2024-04-22T06:17:16.374817+0000","last_unstale":"2024-04-22T06:17:17.266268+0000","last_undegraded":"2024-04-22T06:17:17.266268+0000","last_fullsized":"2024-04-22T06:17:17.266268+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.272095+0000","last_change":"2024-04-22T06:17:16.342100+0000","last_active":"2024-04-22T06:17:18.272095+0000","last_peered":"2024-04-22T06:17:18.272095+0000","last_clean":"2024-04-22T06:17:18.272095+0000","last_became_active":"2024-04-22T06:17:16.339266+0000","last_became_peered":"2024-04-22T06:17:16.339266+0000","last_unstale":"2024-04-22T06:17:18.272095+0000","last_undegraded":"2024-04-22T06:17:18.272095+0000","last_fullsized":"2024-04-22T06:17:18.272095+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'96","reported_seq":138,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:17.378906+0000","last_change":"2024-04-22T06:17:16.310856+0000","last_active":"2024-04-22T06:17:17.378906+0000","last_peered":"2024-04-22T06:17:17.378906+0000","last_clean":"2024-04-22T06:17:17.378906+0000","last_became_active":"2024-04-22T06:17:16.306067+0000","last_became_peered":"2024-04-22T06:17:16.306067+0000","last_unstale":"2024-04-22T06:17:17.378906+0000","last_undegraded":"2024-04-22T06:17:17.378906+0000","last_fullsized":"2024-04-22T06:17:17.378906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.272616+0000","last_change":"2024-04-22T06:17:16.319392+0000","last_active":"2024-04-22T06:17:18.272616+0000","last_peered":"2024-04-22T06:17:18.272616+0000","last_clean":"2024-04-22T06:17:18.272616+0000","last_became_active":"2024-04-22T06:17:16.316499+0000","last_became_peered":"2024-04-22T06:17:16.316499+0000","last_unstale":"2024-04-22T06:17:18.272616+0000","last_undegraded":"2024-04-22T06:17:18.272616+0000","last_fullsized":"2024-04-22T06:17:18.272616+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-22T06:17:16.308561+0000","last_change":"2024-04-22T06:17:16.309062+0000","last_active":"2024-04-22T06:17:16.308561+0000","last_peered":"2024-04-22T06:17:16.308561+0000","last_clean":"2024-04-22T06:17:15.226434+0000","last_became_active":"2024-04-22T06:17:16.297057+0000","last_became_peered":"2024-04-22T06:17:16.297057+0000","last_unstale":"2024-04-22T06:17:16.308561+0000","last_undegraded":"2024-04-22T06:17:16.308561+0000","last_fullsized":"2024-04-22T06:17:16.308561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:17.264188+0000","last_change":"2024-04-22T06:17:16.346365+0000","last_active":"2024-04-22T06:17:17.264188+0000","last_peered":"2024-04-22T06:17:17.264188+0000","last_clean":"2024-04-22T06:17:17.264188+0000","last_became_active":"2024-04-22T06:17:16.341358+0000","last_became_peered":"2024-04-22T06:17:16.341358+0000","last_unstale":"2024-04-22T06:17:17.264188+0000","last_undegraded":"2024-04-22T06:17:17.264188+0000","last_fullsized":"2024-04-22T06:17:17.264188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T06:17:18.269969+0000","last_change":"2024-04-22T06:17:16.360340+0000","last_active":"2024-04-22T06:17:18.269969+0000","last_peered":"2024-04-22T06:17:18.269969+0000","last_clean":"2024-04-22T06:17:18.269969+0000","last_became_active":"2024-04-22T06:17:16.355766+0000","last_became_peered":"2024-04-22T06:17:16.355766+0000","last_unstale":"2024-04-22T06:17:18.269969+0000","last_undegraded":"2024-04-22T06:17:18.269969+0000","last_fullsized":"2024-04-22T06:17:18.269969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T06:17:15.226434+0000","last_clean_scrub_stamp":"2024-04-22T06:17:15.226434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614505,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614505,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":20874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":29970,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T06:17:24.111 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T06:17:24.111 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T06:17:24.111 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T06:17:24.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T06:17:24.217 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:24.215+0000 7f653eced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:24.219 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:17:24.219+0000 7f653eced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:17:24.453 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:24.453 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T06:17:24.466 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T06:17:24.466 INFO:teuthology.run_tasks:Running task cbt... 2024-04-22T06:17:24.480 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi172.front.sepia.ceph.com', 'osds': ['ubuntu@smithi172.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi172.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi172.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi172.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-22T06:17:24.480 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-22T06:17:24.498 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-04-22T06:17:24.498 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-22T06:17:24.551 DEBUG:teuthology.orchestra.run.smithi172:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-22T06:17:24.606 INFO:teuthology.orchestra.run.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-22T06:17:25.170 INFO:tasks.cbt:Installing collectl 2024-04-22T06:17:25.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-22T06:17:25.254 INFO:teuthology.orchestra.run.smithi172.stderr:--2024-04-22 06:17:25-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-22T06:17:25.383 INFO:teuthology.orchestra.run.smithi172.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-04-22T06:17:25.393 INFO:teuthology.orchestra.run.smithi172.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-04-22T06:17:25.602 INFO:teuthology.orchestra.run.smithi172.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-22T06:17:25.602 INFO:teuthology.orchestra.run.smithi172.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmJgD11wq7866YK663dZ7_AcGav2ixu8CK4xqOHw6liC3tF2aO4qnciwJEgZ0k0q8sZRnpImVjdJF0TC65AMq9AHpBNg%3D%3D&use_mirror=netactuate&r= [following] 2024-04-22T06:17:25.602 INFO:teuthology.orchestra.run.smithi172.stderr:--2024-04-22 06:17:25-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmJgD11wq7866YK663dZ7_AcGav2ixu8CK4xqOHw6liC3tF2aO4qnciwJEgZ0k0q8sZRnpImVjdJF0TC65AMq9AHpBNg%3D%3D&use_mirror=netactuate&r= 2024-04-22T06:17:25.613 INFO:teuthology.orchestra.run.smithi172.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-22T06:17:25.696 INFO:teuthology.orchestra.run.smithi172.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-22T06:17:26.002 INFO:teuthology.orchestra.run.smithi172.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-22T06:17:26.002 INFO:teuthology.orchestra.run.smithi172.stderr:Location: https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-22T06:17:26.002 INFO:teuthology.orchestra.run.smithi172.stderr:--2024-04-22 06:17:26-- https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-22T06:17:26.015 INFO:teuthology.orchestra.run.smithi172.stderr:Resolving netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)... 104.225.3.66 2024-04-22T06:17:26.028 INFO:teuthology.orchestra.run.smithi172.stderr:Connecting to netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)|104.225.3.66|:443... connected. 2024-04-22T06:17:26.148 INFO:teuthology.orchestra.run.smithi172.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T06:17:26.148 INFO:teuthology.orchestra.run.smithi172.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-22T06:17:26.148 INFO:teuthology.orchestra.run.smithi172.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-22T06:17:26.148 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:17:26.160 INFO:teuthology.orchestra.run.smithi172.stderr: 0K .......... .......... .......... .......... .......... 8% 3.87M 0s 2024-04-22T06:17:26.186 INFO:teuthology.orchestra.run.smithi172.stderr: 50K .......... .......... .......... .......... .......... 16% 1.94M 0s 2024-04-22T06:17:26.186 INFO:teuthology.orchestra.run.smithi172.stderr: 100K .......... .......... .......... .......... .......... 24% 244M 0s 2024-04-22T06:17:26.199 INFO:teuthology.orchestra.run.smithi172.stderr: 150K .......... .......... .......... .......... .......... 32% 3.91M 0s 2024-04-22T06:17:26.199 INFO:teuthology.orchestra.run.smithi172.stderr: 200K .......... .......... .......... .......... .......... 41% 373M 0s 2024-04-22T06:17:26.200 INFO:teuthology.orchestra.run.smithi172.stderr: 250K .......... .......... .......... .......... .......... 49% 351M 0s 2024-04-22T06:17:26.200 INFO:teuthology.orchestra.run.smithi172.stderr: 300K .......... .......... .......... .......... .......... 57% 206M 0s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 350K .......... .......... .......... .......... .......... 65% 3.91M 0s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 400K .......... .......... .......... .......... .......... 74% 384M 0s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 450K .......... .......... .......... .......... .......... 82% 339M 0s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 500K .......... .......... .......... .......... .......... 90% 405M 0s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 550K .......... .......... .......... .......... .......... 98% 363M 0s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 600K ...... 100% 11.4T=0.06s 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22 06:17:26 (9.25 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-22T06:17:26.214 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/ 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/collectl.conf 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/ 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Socket.html 2024-04-22T06:17:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/colmux.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Import.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/style.css 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Process.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-22T06:17:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Export.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Startup.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Performance.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Messages.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Logging.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-22T06:17:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Misc.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Examples.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Data.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-22T06:17:26.246 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Playback.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Hello.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Network.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Memory.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/Disks.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/service/ 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/service/collectl.service 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/initd/ 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-22T06:17:26.247 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/initd/collectl 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/misc.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/gexpr.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/statsd.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/envrules.std 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/hello.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/vmsum.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/proctree.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/README-WINDOWS 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/formatit.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/colmux 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/INSTALL 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/graphite.ph 2024-04-22T06:17:26.248 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/vmstat.ph 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/README 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/man1/ 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/man1/collectl.1 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/man1/colmux.1 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/lexpr.ph 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/client.pl 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/collectl 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/COPYING 2024-04-22T06:17:26.250 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/vnet.ph 2024-04-22T06:17:26.251 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/GPL 2024-04-22T06:17:26.251 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/UNINSTALL 2024-04-22T06:17:26.251 INFO:teuthology.orchestra.run.smithi172.stdout:collectl-4.3.1/ARTISTIC 2024-04-22T06:17:26.296 INFO:teuthology.orchestra.run.smithi172.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-22T06:17:26.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-22T06:17:26.821 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:21 ago on Mon 22 Apr 2024 06:14:05 AM UTC. 2024-04-22T06:17:26.903 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-22T06:17:26.905 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-22T06:17:26.950 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-04-22T06:17:26.951 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:17:26.951 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-04-22T06:17:26.951 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:17:26.951 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-04-22T06:17:26.951 INFO:teuthology.orchestra.run.smithi172.stdout: librbd-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 23 k 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout: perf x86_64 5.14.0-437.el9 appstream 2.2 M 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout: libradospp-devel x86_64 2:19.0.0-3189.g8d82defa.el9 ceph 30 k 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout:Install 5 Packages 2024-04-22T06:17:26.952 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.4 M 2024-04-22T06:17:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 11 M 2024-04-22T06:17:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-04-22T06:17:27.539 INFO:teuthology.orchestra.run.smithi172.stdout:(1/5): libradospp-devel-19.0.0-3189.g8d82defa.e 69 kB/s | 30 kB 00:00 2024-04-22T06:17:27.614 INFO:teuthology.orchestra.run.smithi172.stdout:(2/5): librbd-devel-19.0.0-3189.g8d82defa.el9.x 45 kB/s | 23 kB 00:00 2024-04-22T06:17:27.797 INFO:teuthology.orchestra.run.smithi172.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 72 kB/s | 13 kB 00:00 2024-04-22T06:17:27.956 INFO:teuthology.orchestra.run.smithi172.stdout:(4/5): perf-5.14.0-437.el9.x86_64.rpm 2.6 MB/s | 2.2 MB 00:00 2024-04-22T06:17:27.981 INFO:teuthology.orchestra.run.smithi172.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 268 kB/s | 118 kB 00:00 2024-04-22T06:17:27.983 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-04-22T06:17:27.983 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.3 MB/s | 2.4 MB 00:01 2024-04-22T06:17:28.013 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-04-22T06:17:28.032 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-04-22T06:17:28.032 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-04-22T06:17:28.141 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-04-22T06:17:28.142 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-04-22T06:17:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-04-22T06:17:28.509 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-22T06:17:28.539 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-22T06:17:28.570 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradospp-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 3/5 2024-04-22T06:17:28.881 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librbd-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 4/5 2024-04-22T06:17:28.957 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : perf-5.14.0-437.el9.x86_64 5/5 2024-04-22T06:17:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: perf-5.14.0-437.el9.x86_64 5/5 2024-04-22T06:17:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradospp-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 1/5 2024-04-22T06:17:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2/5 2024-04-22T06:17:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : perf-5.14.0-437.el9.x86_64 3/5 2024-04-22T06:17:29.797 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: libradospp-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: librbd-devel-2:19.0.0-3189.g8d82defa.el9.x86_64 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: pdsh-2.34-7.el9.x86_64 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: perf-5.14.0-437.el9.x86_64 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:30.063 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-04-22T06:17:30.438 INFO:tasks.cbt:benchmark: librbdfio 2024-04-22T06:17:30.438 DEBUG:teuthology.orchestra.run.smithi172:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-04-22T06:17:30.453 INFO:teuthology.orchestra.run.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-04-22T06:17:34.344 DEBUG:teuthology.orchestra.run.smithi172:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-04-22T06:17:34.849 INFO:teuthology.orchestra.run.smithi172.stdout:Operating system Linux 2024-04-22T06:17:34.849 INFO:teuthology.orchestra.run.smithi172.stdout:CPU x86_64 2024-04-22T06:17:34.850 INFO:teuthology.orchestra.run.smithi172.stdout:Big endian no 2024-04-22T06:17:34.850 INFO:teuthology.orchestra.run.smithi172.stdout:Compiler gcc 2024-04-22T06:17:34.850 INFO:teuthology.orchestra.run.smithi172.stdout:Cross compile no 2024-04-22T06:17:34.850 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-04-22T06:17:34.850 INFO:teuthology.orchestra.run.smithi172.stdout:Static build no 2024-04-22T06:17:34.933 INFO:teuthology.orchestra.run.smithi172.stdout:Wordsize 64 2024-04-22T06:17:34.940 INFO:teuthology.orchestra.run.smithi172.stdout:zlib no 2024-04-22T06:17:34.985 INFO:teuthology.orchestra.run.smithi172.stdout:fcntl(F_FULLFSYNC) no 2024-04-22T06:17:35.000 INFO:teuthology.orchestra.run.smithi172.stdout:Linux AIO support no 2024-04-22T06:17:35.000 INFO:teuthology.orchestra.run.smithi172.stdout:Linux AIO support rw flags no 2024-04-22T06:17:35.081 INFO:teuthology.orchestra.run.smithi172.stdout:POSIX AIO support yes 2024-04-22T06:17:35.081 INFO:teuthology.orchestra.run.smithi172.stdout:POSIX AIO support needs -lrt no 2024-04-22T06:17:35.112 INFO:teuthology.orchestra.run.smithi172.stdout:POSIX AIO fsync yes 2024-04-22T06:17:35.142 INFO:teuthology.orchestra.run.smithi172.stdout:POSIX pshared support yes 2024-04-22T06:17:35.180 INFO:teuthology.orchestra.run.smithi172.stdout:pthread_condattr_setclock() yes 2024-04-22T06:17:35.220 INFO:teuthology.orchestra.run.smithi172.stdout:pthread_sigmask() yes 2024-04-22T06:17:35.257 INFO:teuthology.orchestra.run.smithi172.stdout:pthread_getaffinity_np() yes 2024-04-22T06:17:35.267 INFO:teuthology.orchestra.run.smithi172.stdout:Solaris AIO support no 2024-04-22T06:17:35.322 INFO:teuthology.orchestra.run.smithi172.stdout:__sync_fetch_and_add yes 2024-04-22T06:17:35.370 INFO:teuthology.orchestra.run.smithi172.stdout:__sync_synchronize yes 2024-04-22T06:17:35.399 INFO:teuthology.orchestra.run.smithi172.stdout:__sync_val_compare_and_swap yes 2024-04-22T06:17:35.406 INFO:teuthology.orchestra.run.smithi172.stdout:libverbs no 2024-04-22T06:17:35.420 INFO:teuthology.orchestra.run.smithi172.stdout:rdmacm no 2024-04-22T06:17:35.428 INFO:teuthology.orchestra.run.smithi172.stdout:librpma no 2024-04-22T06:17:35.438 INFO:teuthology.orchestra.run.smithi172.stdout:libprotobuf_c no 2024-04-22T06:17:35.468 INFO:teuthology.orchestra.run.smithi172.stdout:asprintf() yes 2024-04-22T06:17:35.498 INFO:teuthology.orchestra.run.smithi172.stdout:vasprintf() yes 2024-04-22T06:17:35.529 INFO:teuthology.orchestra.run.smithi172.stdout:Linux fallocate yes 2024-04-22T06:17:35.560 INFO:teuthology.orchestra.run.smithi172.stdout:POSIX fadvise yes 2024-04-22T06:17:35.591 INFO:teuthology.orchestra.run.smithi172.stdout:POSIX fallocate yes 2024-04-22T06:17:35.621 INFO:teuthology.orchestra.run.smithi172.stdout:sched_setaffinity(3 arg) yes 2024-04-22T06:17:35.621 INFO:teuthology.orchestra.run.smithi172.stdout:sched_setaffinity(2 arg) no 2024-04-22T06:17:35.652 INFO:teuthology.orchestra.run.smithi172.stdout:clock_gettime yes 2024-04-22T06:17:35.682 INFO:teuthology.orchestra.run.smithi172.stdout:CLOCK_MONOTONIC yes 2024-04-22T06:17:35.713 INFO:teuthology.orchestra.run.smithi172.stdout:clockid_t yes 2024-04-22T06:17:35.745 INFO:teuthology.orchestra.run.smithi172.stdout:gettimeofday yes 2024-04-22T06:17:35.776 INFO:teuthology.orchestra.run.smithi172.stdout:fdatasync yes 2024-04-22T06:17:35.807 INFO:teuthology.orchestra.run.smithi172.stdout:pipe() yes 2024-04-22T06:17:35.837 INFO:teuthology.orchestra.run.smithi172.stdout:pipe2() yes 2024-04-22T06:17:35.868 INFO:teuthology.orchestra.run.smithi172.stdout:pread() yes 2024-04-22T06:17:35.962 INFO:teuthology.orchestra.run.smithi172.stdout:sync_file_range yes 2024-04-22T06:17:35.968 INFO:teuthology.orchestra.run.smithi172.stdout:ASharedMemory_create no 2024-04-22T06:17:35.980 INFO:teuthology.orchestra.run.smithi172.stdout:EXT4 move extent yes 2024-04-22T06:17:36.012 INFO:teuthology.orchestra.run.smithi172.stdout:Linux splice(2) yes 2024-04-22T06:17:36.020 INFO:teuthology.orchestra.run.smithi172.stdout:libnuma no 2024-04-22T06:17:36.049 INFO:teuthology.orchestra.run.smithi172.stdout:strsep yes 2024-04-22T06:17:36.079 INFO:teuthology.orchestra.run.smithi172.stdout:strcasestr yes 2024-04-22T06:17:36.093 INFO:teuthology.orchestra.run.smithi172.stdout:strlcat no 2024-04-22T06:17:36.127 INFO:teuthology.orchestra.run.smithi172.stdout:getopt_long_only() yes 2024-04-22T06:17:36.185 INFO:teuthology.orchestra.run.smithi172.stdout:inet_aton yes 2024-04-22T06:17:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:socklen_t yes 2024-04-22T06:17:36.258 INFO:teuthology.orchestra.run.smithi172.stdout:__thread yes 2024-04-22T06:17:36.297 INFO:teuthology.orchestra.run.smithi172.stdout:RUSAGE_THREAD yes 2024-04-22T06:17:36.329 INFO:teuthology.orchestra.run.smithi172.stdout:SCHED_IDLE yes 2024-04-22T06:17:36.360 INFO:teuthology.orchestra.run.smithi172.stdout:TCP_NODELAY yes 2024-04-22T06:17:36.393 INFO:teuthology.orchestra.run.smithi172.stdout:vsock yes 2024-04-22T06:17:36.426 INFO:teuthology.orchestra.run.smithi172.stdout:Net engine window_size yes 2024-04-22T06:17:36.460 INFO:teuthology.orchestra.run.smithi172.stdout:TCP_MAXSEG yes 2024-04-22T06:17:36.489 INFO:teuthology.orchestra.run.smithi172.stdout:RLIMIT_MEMLOCK yes 2024-04-22T06:17:36.524 INFO:teuthology.orchestra.run.smithi172.stdout:pwritev/preadv yes 2024-04-22T06:17:36.558 INFO:teuthology.orchestra.run.smithi172.stdout:pwritev2/preadv2 yes 2024-04-22T06:17:36.594 INFO:teuthology.orchestra.run.smithi172.stdout:IPv6 helpers yes 2024-04-22T06:17:36.609 INFO:teuthology.orchestra.run.smithi172.stdout:http engine no 2024-04-22T06:17:36.751 INFO:teuthology.orchestra.run.smithi172.stdout:Rados engine yes 2024-04-22T06:17:36.910 INFO:teuthology.orchestra.run.smithi172.stdout:Rados Block Device engine yes 2024-04-22T06:17:37.065 INFO:teuthology.orchestra.run.smithi172.stdout:rbd_poll yes 2024-04-22T06:17:37.216 INFO:teuthology.orchestra.run.smithi172.stdout:rbd_invalidate_cache yes 2024-04-22T06:17:37.247 INFO:teuthology.orchestra.run.smithi172.stdout:setvbuf yes 2024-04-22T06:17:37.255 INFO:teuthology.orchestra.run.smithi172.stdout:Gluster API engine no 2024-04-22T06:17:37.270 INFO:teuthology.orchestra.run.smithi172.stdout:s390_z196_facilities no 2024-04-22T06:17:37.270 INFO:teuthology.orchestra.run.smithi172.stdout:HDFS engine no 2024-04-22T06:17:37.314 INFO:teuthology.orchestra.run.smithi172.stdout:MTD yes 2024-04-22T06:17:37.322 INFO:teuthology.orchestra.run.smithi172.stdout:libpmem no 2024-04-22T06:17:37.322 INFO:teuthology.orchestra.run.smithi172.stdout:libpmem1_5 no 2024-04-22T06:17:37.330 INFO:teuthology.orchestra.run.smithi172.stdout:libpmem2 no 2024-04-22T06:17:37.330 INFO:teuthology.orchestra.run.smithi172.stdout:PMDK dev-dax engine no 2024-04-22T06:17:37.330 INFO:teuthology.orchestra.run.smithi172.stdout:PMDK libpmem engine no 2024-04-22T06:17:37.337 INFO:teuthology.orchestra.run.smithi172.stdout:DDN's Infinite Memory Engine no 2024-04-22T06:17:37.337 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi engine no 2024-04-22T06:17:37.337 INFO:teuthology.orchestra.run.smithi172.stdout:NBD engine no 2024-04-22T06:17:37.346 INFO:teuthology.orchestra.run.smithi172.stdout:DAOS File System (dfs) Engine no 2024-04-22T06:17:37.355 INFO:teuthology.orchestra.run.smithi172.stdout:NFS engine no 2024-04-22T06:17:37.355 INFO:teuthology.orchestra.run.smithi172.stdout:lex/yacc for arithmetic no 2024-04-22T06:17:37.386 INFO:teuthology.orchestra.run.smithi172.stdout:getmntent yes 2024-04-22T06:17:37.426 INFO:teuthology.orchestra.run.smithi172.stdout:getmntinfo no 2024-04-22T06:17:37.476 INFO:teuthology.orchestra.run.smithi172.stdout:Static Assert yes 2024-04-22T06:17:37.503 INFO:teuthology.orchestra.run.smithi172.stdout:bool yes 2024-04-22T06:17:37.536 INFO:teuthology.orchestra.run.smithi172.stdout:strndup yes 2024-04-22T06:17:37.543 INFO:teuthology.orchestra.run.smithi172.stdout:Valgrind headers no 2024-04-22T06:17:37.572 INFO:teuthology.orchestra.run.smithi172.stdout:Zoned block device support yes 2024-04-22T06:17:37.600 INFO:teuthology.orchestra.run.smithi172.stdout:Zoned block device capacity yes 2024-04-22T06:17:37.607 INFO:teuthology.orchestra.run.smithi172.stdout:libzbc engine no 2024-04-22T06:17:37.641 INFO:teuthology.orchestra.run.smithi172.stdout:NVMe uring command support yes 2024-04-22T06:17:37.655 INFO:teuthology.orchestra.run.smithi172.stdout:xnvme engine no 2024-04-22T06:17:37.662 INFO:teuthology.orchestra.run.smithi172.stdout:isal no 2024-04-22T06:17:37.677 INFO:teuthology.orchestra.run.smithi172.stdout:libblkio engine no 2024-04-22T06:17:37.677 INFO:teuthology.orchestra.run.smithi172.stdout:march_armv8_a_crc_crypto no 2024-04-22T06:17:37.677 INFO:teuthology.orchestra.run.smithi172.stdout:cuda no 2024-04-22T06:17:37.677 INFO:teuthology.orchestra.run.smithi172.stdout:libcufile no 2024-04-22T06:17:37.705 INFO:teuthology.orchestra.run.smithi172.stdout:Build march=native yes 2024-04-22T06:17:37.712 INFO:teuthology.orchestra.run.smithi172.stdout:CUnit no 2024-04-22T06:17:37.740 INFO:teuthology.orchestra.run.smithi172.stdout:__kernel_rwf_t yes 2024-04-22T06:17:37.767 INFO:teuthology.orchestra.run.smithi172.stdout:-Wimplicit-fallthrough=2 yes 2024-04-22T06:17:37.797 INFO:teuthology.orchestra.run.smithi172.stdout:-Wno-stringop-truncation yes 2024-04-22T06:17:37.831 INFO:teuthology.orchestra.run.smithi172.stdout:MADV_HUGEPAGE yes 2024-04-22T06:17:37.863 INFO:teuthology.orchestra.run.smithi172.stdout:gettid yes 2024-04-22T06:17:37.896 INFO:teuthology.orchestra.run.smithi172.stdout:statx(2)/libc yes 2024-04-22T06:17:37.931 INFO:teuthology.orchestra.run.smithi172.stdout:statx(2)/syscall yes 2024-04-22T06:17:37.933 INFO:teuthology.orchestra.run.smithi172.stdout:Windows PDB generation no 2024-04-22T06:17:37.964 INFO:teuthology.orchestra.run.smithi172.stdout:timerfd_create yes 2024-04-22T06:17:37.966 INFO:teuthology.orchestra.run.smithi172.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-04-22T06:17:37.966 INFO:teuthology.orchestra.run.smithi172.stdout:Lib-based ioengines dynamic no 2024-04-22T06:17:38.028 INFO:teuthology.orchestra.run.smithi172.stdout:TCMalloc support yes 2024-04-22T06:17:38.029 INFO:teuthology.orchestra.run.smithi172.stdout:seed_buckets 4 2024-04-22T06:17:38.111 INFO:teuthology.orchestra.run.smithi172.stderr:FIO_VERSION = fio-3.37-15-gc948 2024-04-22T06:17:38.145 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc16.o 2024-04-22T06:17:38.266 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc32.o 2024-04-22T06:17:38.303 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc32c-arm64.o 2024-04-22T06:17:38.378 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc32c-intel.o 2024-04-22T06:17:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc32c.o 2024-04-22T06:17:38.463 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc64.o 2024-04-22T06:17:38.504 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crc7.o 2024-04-22T06:17:38.537 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/crct10dif_common.o 2024-04-22T06:17:38.570 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/fnv.o 2024-04-22T06:17:38.612 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/md5.o 2024-04-22T06:17:38.717 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/murmur3.o 2024-04-22T06:17:38.760 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/sha1.o 2024-04-22T06:17:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/sha256.o 2024-04-22T06:17:39.228 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/sha3.o 2024-04-22T06:17:39.377 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/sha512.o 2024-04-22T06:17:39.490 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/test.o 2024-04-22T06:17:39.652 INFO:teuthology.orchestra.run.smithi172.stdout: CC crc/xxhash.o 2024-04-22T06:17:39.748 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/axmap.o 2024-04-22T06:17:39.872 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/bloom.o 2024-04-22T06:17:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/flist_sort.o 2024-04-22T06:17:40.027 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/gauss.o 2024-04-22T06:17:40.115 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/getrusage.o 2024-04-22T06:17:40.151 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/hweight.o 2024-04-22T06:17:40.189 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/ieee754.o 2024-04-22T06:17:40.240 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/lfsr.o 2024-04-22T06:17:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/memalign.o 2024-04-22T06:17:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/memcpy.o 2024-04-22T06:17:40.541 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/mountcheck.o 2024-04-22T06:17:40.581 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/num2str.o 2024-04-22T06:17:40.648 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/output_buffer.o 2024-04-22T06:17:40.696 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/pattern.o 2024-04-22T06:17:40.842 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/prio_tree.o 2024-04-22T06:17:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/rand.o 2024-04-22T06:17:41.149 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/rbtree.o 2024-04-22T06:17:41.275 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/strntol.o 2024-04-22T06:17:41.323 INFO:teuthology.orchestra.run.smithi172.stdout: CC lib/zipf.o 2024-04-22T06:17:41.445 INFO:teuthology.orchestra.run.smithi172.stdout: CC gettime.o 2024-04-22T06:17:41.735 INFO:teuthology.orchestra.run.smithi172.stdout: CC ioengines.o 2024-04-22T06:17:41.997 INFO:teuthology.orchestra.run.smithi172.stdout: CC init.o 2024-04-22T06:17:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: CC stat.o 2024-04-22T06:17:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: CC log.o 2024-04-22T06:17:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: CC time.o 2024-04-22T06:17:44.888 INFO:teuthology.orchestra.run.smithi172.stdout: CC filesetup.o 2024-04-22T06:17:45.585 INFO:teuthology.orchestra.run.smithi172.stdout: CC eta.o 2024-04-22T06:17:45.900 INFO:teuthology.orchestra.run.smithi172.stdout: CC verify.o 2024-04-22T06:17:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: CC memory.o 2024-04-22T06:17:46.766 INFO:teuthology.orchestra.run.smithi172.stdout: CC io_u.o 2024-04-22T06:17:47.549 INFO:teuthology.orchestra.run.smithi172.stdout: CC parse.o 2024-04-22T06:17:47.998 INFO:teuthology.orchestra.run.smithi172.stdout: CC fio_sem.o 2024-04-22T06:17:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: CC rwlock.o 2024-04-22T06:17:48.189 INFO:teuthology.orchestra.run.smithi172.stdout: CC pshared.o 2024-04-22T06:17:48.255 INFO:teuthology.orchestra.run.smithi172.stdout: CC options.o 2024-04-22T06:17:49.050 INFO:teuthology.orchestra.run.smithi172.stdout: CC smalloc.o 2024-04-22T06:17:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: CC filehash.o 2024-04-22T06:17:49.394 INFO:teuthology.orchestra.run.smithi172.stdout: CC profile.o 2024-04-22T06:17:49.511 INFO:teuthology.orchestra.run.smithi172.stdout: CC debug.o 2024-04-22T06:17:49.554 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/cpu.o 2024-04-22T06:17:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/mmap.o 2024-04-22T06:17:49.864 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/sync.o 2024-04-22T06:17:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/null.o 2024-04-22T06:17:50.212 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/net.o 2024-04-22T06:17:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/ftruncate.o 2024-04-22T06:17:50.749 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/fileoperations.o 2024-04-22T06:17:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/exec.o 2024-04-22T06:17:51.063 INFO:teuthology.orchestra.run.smithi172.stdout: CC server.o 2024-04-22T06:17:51.807 INFO:teuthology.orchestra.run.smithi172.stdout: CC client.o 2024-04-22T06:17:52.355 INFO:teuthology.orchestra.run.smithi172.stdout: CC iolog.o 2024-04-22T06:17:52.798 INFO:teuthology.orchestra.run.smithi172.stdout: CC backend.o 2024-04-22T06:17:53.659 INFO:teuthology.orchestra.run.smithi172.stdout: CC libfio.o 2024-04-22T06:17:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: CC flow.o 2024-04-22T06:17:53.947 INFO:teuthology.orchestra.run.smithi172.stdout: CC cconv.o 2024-04-22T06:17:54.301 INFO:teuthology.orchestra.run.smithi172.stdout: CC gettime-thread.o 2024-04-22T06:17:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: CC helpers.o 2024-04-22T06:17:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: CC json.o 2024-04-22T06:17:54.650 INFO:teuthology.orchestra.run.smithi172.stdout: CC idletime.o 2024-04-22T06:17:54.878 INFO:teuthology.orchestra.run.smithi172.stdout: CC td_error.o 2024-04-22T06:17:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: CC profiles/tiobench.o 2024-04-22T06:17:55.073 INFO:teuthology.orchestra.run.smithi172.stdout: CC profiles/act.o 2024-04-22T06:17:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: CC io_u_queue.o 2024-04-22T06:17:55.283 INFO:teuthology.orchestra.run.smithi172.stdout: CC filelock.o 2024-04-22T06:17:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: CC workqueue.o 2024-04-22T06:17:55.586 INFO:teuthology.orchestra.run.smithi172.stdout: CC rate-submit.o 2024-04-22T06:17:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: CC optgroup.o 2024-04-22T06:17:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: CC helper_thread.o 2024-04-22T06:17:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: CC steadystate.o 2024-04-22T06:17:56.207 INFO:teuthology.orchestra.run.smithi172.stdout: CC zone-dist.o 2024-04-22T06:17:56.346 INFO:teuthology.orchestra.run.smithi172.stdout: CC zbd.o 2024-04-22T06:17:57.059 INFO:teuthology.orchestra.run.smithi172.stdout: CC dedupe.o 2024-04-22T06:17:57.204 INFO:teuthology.orchestra.run.smithi172.stdout: CC fdp.o 2024-04-22T06:17:57.350 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/posixaio.o 2024-04-22T06:17:57.484 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/falloc.o 2024-04-22T06:17:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/e4defrag.o 2024-04-22T06:17:57.727 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/splice.o 2024-04-22T06:17:57.918 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/asprintf.o 2024-04-22T06:17:57.953 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/strlcat.o 2024-04-22T06:17:57.996 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/mtd.o 2024-04-22T06:17:58.158 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/libmtd.o 2024-04-22T06:17:58.540 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/libmtd_legacy.o 2024-04-22T06:17:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: CC diskutil.o 2024-04-22T06:17:58.880 INFO:teuthology.orchestra.run.smithi172.stdout: CC fifo.o 2024-04-22T06:17:58.936 INFO:teuthology.orchestra.run.smithi172.stdout: CC blktrace.o 2024-04-22T06:17:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: CC cgroup.o 2024-04-22T06:17:59.421 INFO:teuthology.orchestra.run.smithi172.stdout: CC trim.o 2024-04-22T06:17:59.535 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/sg.o 2024-04-22T06:17:59.918 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/linux-dev-lookup.o 2024-04-22T06:17:59.975 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/io_uring.o 2024-04-22T06:18:00.362 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/nvme.o 2024-04-22T06:18:00.676 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/linux-blkzoned.o 2024-04-22T06:18:00.833 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/cmdprio.o 2024-04-22T06:18:01.074 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/rados.o 2024-04-22T06:18:01.247 INFO:teuthology.orchestra.run.smithi172.stdout: CC engines/rbd.o 2024-04-22T06:18:01.434 INFO:teuthology.orchestra.run.smithi172.stdout: CC fio.o 2024-04-22T06:18:01.523 INFO:teuthology.orchestra.run.smithi172.stdout: LINK fio 2024-04-22T06:18:01.766 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/log.o 2024-04-22T06:18:01.807 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/genzipf.o 2024-04-22T06:18:01.947 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/strcasestr.o 2024-04-22T06:18:01.971 INFO:teuthology.orchestra.run.smithi172.stdout: CC oslib/strndup.o 2024-04-22T06:18:01.994 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/fio-genzipf 2024-04-22T06:18:02.132 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/btrace2fio.o 2024-04-22T06:18:02.558 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/fio-btrace2fio 2024-04-22T06:18:02.697 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/verify-state.o 2024-04-22T06:18:02.795 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/debug.o 2024-04-22T06:18:02.831 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/fio-verify-state 2024-04-22T06:18:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/stest.o 2024-04-22T06:18:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/arch.o 2024-04-22T06:18:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/stest 2024-04-22T06:18:03.209 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/ieee754.o 2024-04-22T06:18:03.249 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/ieee754 2024-04-22T06:18:03.388 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/axmap.o 2024-04-22T06:18:03.489 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/axmap 2024-04-22T06:18:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/lfsr-test.o 2024-04-22T06:18:03.726 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/lfsr-test 2024-04-22T06:18:03.866 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/gen-rand.o 2024-04-22T06:18:03.963 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/gen-rand 2024-04-22T06:18:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/memlock.o 2024-04-22T06:18:04.160 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/memlock 2024-04-22T06:18:04.302 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/read-to-pipe-async.o 2024-04-22T06:18:04.495 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/read-to-pipe-async 2024-04-22T06:18:04.631 INFO:teuthology.orchestra.run.smithi172.stdout: CC t/io_uring.o 2024-04-22T06:18:05.167 INFO:teuthology.orchestra.run.smithi172.stdout: LINK t/io_uring 2024-04-22T06:18:05.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-22T06:18:05.335 DEBUG:teuthology.orchestra.run.smithi172:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr:06:18:05 - DEBUG - cbt - Settings.cluster: 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: 'clients': ['ubuntu@smithi172.front.sepia.ceph.com'], 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: 'head': 'ubuntu@smithi172.front.sepia.ceph.com', 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: 'is_teuthology': True, 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: 'iterations': 1, 2024-04-22T06:18:05.566 INFO:teuthology.orchestra.run.smithi172.stderr: 'mons': ['ubuntu@smithi172.front.sepia.ceph.com'], 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'osds': ['ubuntu@smithi172.front.sepia.ceph.com'], 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'osds_per_node': 3, 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'pid_dir': '/var/run/ceph', 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'rebuild_every_test': False, 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'rgws': {'ubuntu@smithi172.front.sepia.ceph.com': None}, 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'use_existing': True, 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr: 'user': 'ubuntu'} 2024-04-22T06:18:05.567 INFO:teuthology.orchestra.run.smithi172.stderr:06:18:05 - DEBUG - cbt - Nodes : ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:18:05.569 INFO:teuthology.orchestra.run.smithi172.stderr:06:18:05 - DEBUG - cbt - Nodes : ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:18:05.569 INFO:teuthology.orchestra.run.smithi172.stderr:06:18:05 - DEBUG - cbt - get_fqdn_local()=smithi172 2024-04-22T06:18:05.569 INFO:teuthology.orchestra.run.smithi172.stderr:06:18:05 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@smithi172.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi 2024-04-22T06:18:05.686 INFO:teuthology.orchestra.run.smithi172.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr:Traceback (most recent call last): 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 93, in 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: exit(main(sys.argv)) 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 64, in main 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: b.initialize() 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/librbdfio.py", line 73, in initialize 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: common.clean_remote_dir(self.run_dir) 2024-04-22T06:18:05.687 INFO:teuthology.orchestra.run.smithi172.stderr: File "/home/ubuntu/cephtest/cbt/common.py", line 232, in clean_remote_dir 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr: pdsh(nodes, 'if [ -d "%s" ]; then rm -rf %s; fi' % (remote_dir, remote_dir), 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr: File "/home/ubuntu/cephtest/cbt/common.py", line 55, in communicate 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr: raise Exception('\n'.join([str(self), 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:Exception: checked_Popen args=pdsh -S -f 1 -R ssh -w ubuntu@smithi172.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi continue_if_error=False rtncode=255 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:stdout: 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:stderr: 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:smithi172: Warning: Permanently added 'smithi172.front.sepia.ceph.com' (ECDSA) to the list of known hosts. 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:smithi172: Permission denied, please try again. 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:smithi172: Permission denied, please try again. 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:smithi172: ubuntu@smithi172.front.sepia.ceph.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr:pdsh@smithi172: smithi172: ssh exited with exit code 255 2024-04-22T06:18:05.688 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:18:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:18:05.699 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/cbt.py", line 244, in begin self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi172 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 2024-04-22T06:18:05.909 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2692ff3c9d8a4d63b435d59c187cbef1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_8d82defa854b9197ed2c2b855ac5e996c480ec88/qa/tasks/cbt.py", line 244, in begin self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi172 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 2024-04-22T06:18:05.911 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-22T06:18:06.115 DEBUG:teuthology.orchestra.run.smithi172:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-22T06:18:06.151 DEBUG:teuthology.orchestra.run.smithi172:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-04-22T06:18:06.226 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-22T06:18:06.226 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-22T06:18:06.226 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-22T06:18:06.281 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T06:18:06.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T06:18:06.433 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:18:06.426+0000 7f31f135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:18:06.434 INFO:teuthology.orchestra.run.smithi172.stderr:2024-04-22T06:18:06.435+0000 7f31f135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T06:18:06.642 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T06:18:06.642 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T06:18:06.642 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T06:18:06.642 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:18:06.697 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T06:18:06.697 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T06:18:06.697 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:18:06.764 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T06:18:06.764 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T06:18:06.765 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:18:06.828 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T06:18:06.828 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T06:18:06.828 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T06:18:06.828 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:18:06.854 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T06:18:06.855 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T06:18:06.855 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T06:18:06.855 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T06:18:06.905 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T06:18:06.905 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T06:18:06.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-22T06:18:06.972 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:18:06.972 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T06:18:07.193 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:18:07.194 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T06:18:07.300 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:18:07.300 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T06:18:07.391 INFO:tasks.ceph:Archiving mon data... 2024-04-22T06:18:07.392 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184/data/mon.a.tgz 2024-04-22T06:18:07.392 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-04-22T06:18:07.411 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.KTL3mKo0Nw 2024-04-22T06:18:07.411 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KTL3mKo0Nw 2024-04-22T06:18:07.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.KTL3mKo0Nw 2024-04-22T06:18:07.583 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.KTL3mKo0Nw is 138KB 2024-04-22T06:18:07.609 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.KTL3mKo0Nw 2024-04-22T06:18:07.624 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T06:18:07.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T06:18:07.740 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T06:18:07.741 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T06:18:07.742 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184/remote/smithi172/crash 2024-04-22T06:18:07.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T06:18:07.803 INFO:tasks.ceph:Compressing logs... 2024-04-22T06:18:07.804 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T06:18:07.883 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27852.log 2024-04-22T06:18:07.883 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28013.log 2024-04-22T06:18:07.883 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28961.log 2024-04-22T06:18:07.883 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.27852.log: 50.0%/var/log/ceph/ceph-client.admin.28013.log: -- replaced with /var/log/ceph/ceph-client.admin.27852.log.gz 2024-04-22T06:18:07.883 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28777.log 2024-04-22T06:18:07.883 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28013.log.gz 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28961.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28961.log.gz 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28880.log 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28777.log.gz 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr: 83.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29339.log 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28880.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28880.log.gz 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5/var/log/ceph/ceph-osd.2.log: --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T06:18:07.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28635.log 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29339.log.gz 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28820.log 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.28635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28691.log 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr: 86.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28635.log.gz 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29592.log 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28820.log.gz 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29087.log 2024-04-22T06:18:07.885 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28691.log.gz 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29592.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29592.log.gz 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29087.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29087.log.gz 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29300.log 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29465.log 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29213.log 2024-04-22T06:18:07.886 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29300.log.gz 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29465.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29465.log.gz 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29213.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29213.log.gz 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29174.log 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29088.log 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29174.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29174.log.gz 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr: -- /var/log/ceph/ceph-client.admin.28053.log 2024-04-22T06:18:07.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29088.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29088.log.gz 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29552.log 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28241.log 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28053.log.gz 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29552.log: gzip -5 --verbose -- 48.0% -- replaced with /var/log/ceph/ceph-client.admin.29552.log.gz 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.32921.log 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28241.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28241.log.gz 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28211.log 2024-04-22T06:18:07.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.32921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29762.log 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32921.log.gz 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28211.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28211.log.gz 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.29720.log 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29762.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29762.log.gz 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29678.log 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29720.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26389.log 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29720.log.gz 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29340.log 2024-04-22T06:18:07.889 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29678.log.gz 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28735.log 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.tmp-client.admin.26389.log: 67.2% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26389.log.gz 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29340.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29340.log.gz 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.29466.log 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28735.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28735.log.gz 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27764.log 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28960.log 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29466.log.gz 2024-04-22T06:18:07.890 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.27764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28962.log 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27764.log.gz 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29426.log 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28960.log.gz 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.28962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27966.log 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28962.log.gz 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29426.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29426.log.gz 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr: /var/log/ceph/ceph-client.admin.29591.log 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.27966.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27966.log.gz 2024-04-22T06:18:07.891 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29086.log 2024-04-22T06:18:07.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29214.log 2024-04-22T06:18:07.892 INFO:teuthology.orchestra.run.smithi172.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29591.log.gz 2024-04-22T06:18:07.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29086.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29086.log.gz 2024-04-22T06:18:07.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.29214.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29214.log.gz 2024-04-22T06:18:07.892 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-22T06:18:07.892 INFO:teuthology.orchestra.run.smithi172.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-22T06:18:07.893 INFO:teuthology.orchestra.run.smithi172.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-22T06:18:07.894 INFO:teuthology.orchestra.run.smithi172.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-22T06:18:07.896 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-04-22T06:18:07.896 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m0.038s 2024-04-22T06:18:07.896 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m0.089s 2024-04-22T06:18:07.896 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.031s 2024-04-22T06:18:07.896 INFO:tasks.ceph:Archiving logs... 2024-04-22T06:18:07.896 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184/remote/smithi172/log 2024-04-22T06:18:07.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T06:18:08.044 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T06:18:08.055 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T06:18:08.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T06:18:08.109 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T06:18:08.118 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T06:18:08.119 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T06:18:08.134 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-04-22T06:18:08.138 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T06:18:08.138 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-04-22T06:18:08.138 INFO:teuthology.orchestra.run.smithi172.stdout:^- hv01.front.sepia.ceph.com 2 6 177 59 +1022us[+1022us] +/- 596ms 2024-04-22T06:18:08.138 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv02.front.sepia.ceph.com 3 6 177 59 -263us[ -263us] +/- 75ms 2024-04-22T06:18:08.138 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv03.front.sepia.ceph.com 3 6 177 60 +221us[ +238us] +/- 66ms 2024-04-22T06:18:08.138 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T06:18:08.139 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T06:18:08.147 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T06:18:08.148 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T06:18:08.159 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T06:18:08.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T06:18:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:18:08.260 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T06:18:08.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T06:18:08.312 INFO:teuthology.task.internal:Duration was 468.778573 seconds 2024-04-22T06:18:08.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T06:18:08.359 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T06:18:08.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T06:18:08.396 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T06:18:08.776 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T06:18:08.776 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-04-22T06:18:08.776 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T06:18:08.836 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T06:18:08.836 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T06:18:08.901 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T06:18:08.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T06:18:09.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T06:18:09.043 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T06:18:09.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T06:18:09.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T06:18:09.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:18:09.129 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-04-22T06:18:09.153 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T06:18:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T06:18:09.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T06:18:09.218 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T06:18:09.219 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184/remote/smithi172 2024-04-22T06:18:09.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T06:18:09.283 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T06:18:09.283 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T06:18:09.329 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T06:18:09.340 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T06:18:09.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T06:18:09.355 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T06:18:09.356 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T06:18:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 06:18 /home/ubuntu/cephtest 2024-04-22T06:18:09.384 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T06:18:09.400 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T06:18:09.428 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T06:18:09.461 DEBUG:teuthology.nuke:shortname: smithi172 2024-04-22T06:18:09.461 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T06:18:09.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_05:57:37-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-4-base-distro-default-smithi/7668184', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 06:04:53.289752', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFA0D68EFUFtLZRcE4tO2yC1Fv9tXMjfIvB1Rmx18Iz6fjurTGkPY1y6+HtXjK9pqji3SiR9CmYnGRGG68D1oNo='} 2024-04-22T06:18:09.554 INFO:teuthology.orchestra.console.smithi172:Power off 2024-04-22T06:18:09.555 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T06:18:09.613 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-04-22T06:18:09.613 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:18:09.663 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-04-22T06:18:13.665 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:18:13.776 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-04-22T06:18:17.778 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T06:18:17.892 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-04-22T06:18:17.994 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-04-22T06:18:18.098 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} duration: 468.7785725593567 failure_reason: 'Command failed on smithi172 with status 1: ''/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2692ff3c9d8a4d63b435d59c187cbef1 status: fail success: false 2024-04-22T06:18:18.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T06:18:18.193 INFO:teuthology.run:FAIL