2024-04-02T02:53:17.557 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-02T02:53:17.557 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yingxin@teuthology --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} --archive /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285 --name yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi -- /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285/orig.config.yaml 2024-04-02T02:53:17.604 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T02:53:17.689 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285 branch: ci-yingxin-crimson-load-obc-3-base description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7635285' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-3-base ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: c2d32edc8c5fac8717d9c592e849ca877e2f524e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: c2d32edc8c5fac8717d9c592e849ca877e2f524e selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-3-base env: CRIMSON_COMPAT: '1' sha1: c2d32edc8c5fac8717d9c592e849ca877e2f524e owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: c2d32edc8c5fac8717d9c592e849ca877e2f524e sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-3-base suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c2d32edc8c5fac8717d9c592e849ca877e2f524e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c2d32edc8c5fac8717d9c592e849ca877e2f524e targets: smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKCaDYUlgRjb+fcIIxGdyf1o5fBbkFNRF4sopkMh8EhVgc0uLzz6x1UIVrw/4ANIHPS2pef/KUQ70dpPvNVtOc= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-02_02:43:05 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-02T02:53:17.689 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c2d32edc8c5fac8717d9c592e849ca877e2f524e/qa; will attempt to use it 2024-04-02T02:53:17.690 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c2d32edc8c5fac8717d9c592e849ca877e2f524e/qa/tasks 2024-04-02T02:53:17.690 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-02T02:53:17.692 INFO:teuthology.task.internal:Checking packages... 2024-04-02T02:53:17.718 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'c2d32edc8c5fac8717d9c592e849ca877e2f524e' 2024-04-02T02:53:17.718 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-02T02:53:17.718 INFO:teuthology.packaging:ref: None 2024-04-02T02:53:17.718 INFO:teuthology.packaging:tag: None 2024-04-02T02:53:17.718 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-3-base 2024-04-02T02:53:17.718 INFO:teuthology.packaging:sha1: c2d32edc8c5fac8717d9c592e849ca877e2f524e 2024-04-02T02:53:17.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-3-base 2024-04-02T02:53:17.939 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2461.gc2d32edc 2024-04-02T02:53:17.941 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-02T02:53:17.948 INFO:teuthology.task.internal:no buildpackages task found 2024-04-02T02:53:17.948 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-02T02:53:17.969 INFO:teuthology.task.internal:Saving configuration 2024-04-02T02:53:17.987 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-02T02:53:17.993 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-02T02:53:18.018 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-02 02:48:17.527359', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKCaDYUlgRjb+fcIIxGdyf1o5fBbkFNRF4sopkMh8EhVgc0uLzz6x1UIVrw/4ANIHPS2pef/KUQ70dpPvNVtOc='} 2024-04-02T02:53:18.018 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-02T02:53:18.024 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-02T02:53:18.024 INFO:teuthology.run_tasks:Running task console_log... 2024-04-02T02:53:18.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb91ef46e50>, signals=[15]) 2024-04-02T02:53:18.067 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-02T02:53:18.073 INFO:teuthology.task.internal:Opening connections... 2024-04-02T02:53:18.073 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T02:53:18.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T02:53:18.165 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-02T02:53:18.188 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-02T02:53:18.251 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-02T02:53:18.251 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-02T02:53:18.308 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-02T02:53:18.309 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-02T02:53:18.309 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-02T02:53:18.309 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-02T02:53:18.309 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-02T02:53:18.329 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-02T02:53:18.338 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-02T02:53:18.343 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-02T02:53:18.344 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-02T02:53:18.364 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-02T02:53:18.370 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-02T02:53:18.370 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-02T02:53:18.434 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-02T02:53:18.467 INFO:teuthology.run_tasks:Running task kernel... 2024-04-02T02:53:18.481 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-02T02:53:18.481 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-02T02:53:18.481 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-02T02:53:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T02:53:18.496 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-04-02T02:53:18.553 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-04-02T02:53:18.553 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-04-02T02:53:18.553 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-04-02T02:53:20.605 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-02T02:53:22.272 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-04-02T02:53:26.854 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-04-02T02:53:29.663 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-02T02:53:41.097 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 792 kB/s | 6.1 MB 00:07 2024-04-02T02:53:42.605 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-04-02T02:53:50.709 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 21 MB 00:07 2024-04-02T02:53:55.647 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-02T02:53:57.879 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-02T02:53:57.879 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-02T02:53:57.968 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T02:53:57.975 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T02:53:57.975 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T02:53:57.975 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-04-02T02:53:57.976 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:53:57.977 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 76 M 2024-04-02T02:53:57.977 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-04-02T02:53:57.977 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T02:53:58.190 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 900 kB/s | 192 kB 00:00 2024-04-02T02:53:59.090 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-04-02T02:53:59.507 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 21 MB/s | 28 MB 00:01 2024-04-02T02:53:59.908 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-02T02:53:59.911 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T02:53:59.911 INFO:teuthology.orchestra.run.smithi186.stdout:Total 40 MB/s | 76 MB 00:01 2024-04-02T02:53:59.912 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T02:54:00.297 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T02:54:00.297 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T02:54:01.159 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T02:54:01.159 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T02:54:03.445 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T02:54:04.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-02T02:54:04.182 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T02:54:06.968 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T02:54:07.110 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T02:54:11.814 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T02:54:11.850 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T02:54:16.558 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T02:54:46.202 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T02:54:46.202 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-02T02:54:46.202 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:54:46.245 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-02T02:54:46.632 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-02T02:54:46.632 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-02T02:54:46.632 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-02T02:54:46.632 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:54:47.060 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T02:54:47.396 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T02:54:47.523 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T02:54:47.523 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T02:54:47.523 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-02T02:54:47.523 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-02T02:54:47.523 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-02T02:54:47.523 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-02T02:54:47.523 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-04-02T02:54:47.524 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-04-02T02:54:47.603 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-02T02:54:47.603 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-02T02:54:47.604 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-04-02T02:54:47.645 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-04-02T02:54:47.645 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-04-02T02:54:47.645 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-04-02T02:54:47.645 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-04-02T02:54:47.645 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-04-02T02:54:47.646 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T02:54:47.647 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-04-02T02:54:47.648 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-02T02:54:47.648 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-434.el9.x86_64 2024-04-02T02:54:47.648 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-02T02:54:48.297 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-04-02T02:54:49.589 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-02T02:54:49.618 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-04-02T02:54:49.620 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-02T02:54:49.636 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.ZxbJfHDVBc 2024-04-02T02:54:49.636 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZxbJfHDVBc 2024-04-02T02:54:49.711 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.ZxbJfHDVBc 2024-04-02T02:54:49.855 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.ZxbJfHDVBc is 6KB 2024-04-02T02:54:49.877 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.ZxbJfHDVBc 2024-04-02T02:54:49.892 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-04-02T02:54:49.956 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-02T02:54:49.957 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-02T02:54:49.957 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-02T02:54:49.957 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-02T02:54:49.958 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-02T02:54:50.107 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-04-02T02:55:20.130 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-02T02:55:20.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T02:55:20.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T02:55:38.529 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-02T02:55:47.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T02:55:47.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T02:55:50.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-02T02:56:02.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T02:56:02.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T02:56:21.027 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-02T02:56:36.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T02:56:36.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T02:56:36.918 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-02T02:56:37.394 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T02:56:37.394 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-02T02:56:37.394 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-04-02T02:56:37.409 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-434.el9.x86_64 2024-04-02T02:56:37.409 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-02T02:56:37.409 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-02T02:56:37.409 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-02T02:56:38.411 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-02T02:56:38.411 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-02T02:56:38.528 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-04-02T02:56:38.572 DEBUG:teuthology.parallel:result is None 2024-04-02T02:56:38.572 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-02T02:56:38.579 INFO:teuthology.task.internal:Creating test directory... 2024-04-02T02:56:38.579 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-02T02:56:38.594 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-02T02:56:38.600 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-02T02:56:38.607 INFO:teuthology.task.internal:Creating archive directory... 2024-04-02T02:56:38.607 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-02T02:56:38.654 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-02T02:56:38.661 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-02T02:56:38.661 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-02T02:56:38.749 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T02:56:38.762 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-02T02:56:38.764 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-02T02:56:38.771 INFO:teuthology.task.internal:Configuring sudo... 2024-04-02T02:56:38.771 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-02T02:56:38.799 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-02T02:56:38.809 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-02T02:56:38.809 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-02T02:56:38.854 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T02:56:38.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-02T02:56:38.982 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T02:56:39.058 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-02T02:56:39.109 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T02:56:39.109 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-02T02:56:39.185 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-02T02:56:39.282 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T02:56:39.612 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-02T02:56:39.620 INFO:teuthology.task.internal:Starting timer... 2024-04-02T02:56:39.621 INFO:teuthology.run_tasks:Running task pcp... 2024-04-02T02:56:39.631 INFO:teuthology.run_tasks:Running task selinux... 2024-04-02T02:56:39.640 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-02T02:56:39.640 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-02T02:56:39.683 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-02T02:56:39.685 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-02T02:56:39.685 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-02T02:56:39.741 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-02T02:56:39.741 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-02T02:56:39.742 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T02:56:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T02:56:39.810 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-02T02:56:39.810 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-02T02:56:39.886 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-02T02:56:39.896 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-02T02:56:39.901 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-02T02:56:40.031 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-02T02:56:40.047 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-02T02:56:40.048 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-02T03:00:56.262 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-02T03:00:56.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T03:00:56.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-02T03:00:56.363 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-02T03:00:56.445 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-02T03:00:56.446 INFO:teuthology.run_tasks:Running task clock... 2024-04-02T03:00:56.456 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-02T03:00:56.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-02T03:00:56.457 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T03:00:56.518 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-02T03:00:56.533 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-02T03:00:56.562 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-02T03:00:56.576 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-02T03:00:56.593 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-02T03:00:56.610 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-02T03:00:56.664 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-02T03:00:56.667 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T03:00:56.668 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-02T03:00:56.668 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T03:00:56.668 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T03:00:56.668 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T03:00:56.668 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T03:00:56.669 INFO:teuthology.run_tasks:Running task install... 2024-04-02T03:00:56.677 DEBUG:teuthology.task.install:project ceph 2024-04-02T03:00:56.677 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e'}} 2024-04-02T03:00:56.677 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e'} 2024-04-02T03:00:56.677 INFO:teuthology.task.install:Using flavor: crimson 2024-04-02T03:00:56.683 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-02T03:00:56.683 INFO:teuthology.task.install:extra packages: [] 2024-04-02T03:00:56.684 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e', 'tag': None, 'wait_for_package': False} 2024-04-02T03:00:56.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=c2d32edc8c5fac8717d9c592e849ca877e2f524e 2024-04-02T03:00:56.828 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3-base/c2d32edc8c5fac8717d9c592e849ca877e2f524e/centos/9/flavors/crimson/ 2024-04-02T03:00:56.828 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2461.gc2d32edc 2024-04-02T03:00:56.941 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3-base/c2d32edc8c5fac8717d9c592e849ca877e2f524e/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3-base/c2d32edc8c5fac8717d9c592e849ca877e2f524e/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3-base/c2d32edc8c5fac8717d9c592e849ca877e2f524e/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-02T03:00:56.941 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:00:56.941 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-02T03:00:56.969 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-02T03:00:56.969 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2d32edc8c5fac8717d9c592e849ca877e2f524e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-02T03:00:57.036 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-02T03:00:57.116 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-02T03:00:57.183 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-02T03:00:57.184 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-02T03:00:57.429 INFO:teuthology.orchestra.run.smithi186.stdout:74 files removed 2024-04-02T03:00:57.456 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw 2024-04-02T03:00:58.322 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 155 kB/s | 87 kB 00:00 2024-04-02T03:00:58.797 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-04-02T03:00:59.173 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 6.0 kB/s | 2.0 kB 00:00 2024-04-02T03:00:59.767 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-02T03:01:01.343 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-04-02T03:01:02.342 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-02T03:01:04.643 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-02T03:01:09.727 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-04-02T03:01:11.603 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-02T03:01:13.063 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-02T03:01:18.063 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-02T03:01:20.642 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 9.6 M 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 3.4 M 2024-04-02T03:01:20.652 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 3.7 M 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 5.9 M 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 19 M 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 25 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 796 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-02T03:01:20.653 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 546 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 5.0 M 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 45 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 134 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 159 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 314 k 2024-04-02T03:01:20.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 294 k 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 99 k 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout:Install 30 Packages 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 2 Packages 2024-04-02T03:01:20.655 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:20.656 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 58 M 2024-04-02T03:01:20.656 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:01:23.422 INFO:teuthology.orchestra.run.smithi186.stdout:(1/32): ceph-base-19.0.0-2461.gc2d32edc.el9.x86 2.5 MB/s | 5.9 MB 00:02 2024-04-02T03:01:23.489 INFO:teuthology.orchestra.run.smithi186.stdout:(2/32): ceph-selinux-19.0.0-2461.gc2d32edc.el9. 378 kB/s | 25 kB 00:00 2024-04-02T03:01:23.647 INFO:teuthology.orchestra.run.smithi186.stdout:(3/32): libcephfs2-19.0.0-2461.gc2d32edc.el9.x8 4.9 MB/s | 796 kB 00:00 2024-04-02T03:01:23.773 INFO:teuthology.orchestra.run.smithi186.stdout:(4/32): libradosstriper1-19.0.0-2461.gc2d32edc. 4.3 MB/s | 546 kB 00:00 2024-04-02T03:01:24.098 INFO:teuthology.orchestra.run.smithi186.stdout:(5/32): ceph-radosgw-19.0.0-2461.gc2d32edc.el9. 3.1 MB/s | 9.6 MB 00:03 2024-04-02T03:01:24.290 INFO:teuthology.orchestra.run.smithi186.stdout:(6/32): ceph-common-19.0.0-2461.gc2d32edc.el9.x 5.8 MB/s | 19 MB 00:03 2024-04-02T03:01:24.315 INFO:teuthology.orchestra.run.smithi186.stdout:(7/32): python3-ceph-argparse-19.0.0-2461.gc2d3 208 kB/s | 45 kB 00:00 2024-04-02T03:01:24.341 INFO:teuthology.orchestra.run.smithi186.stdout:(8/32): python3-ceph-common-19.0.0-2461.gc2d32e 2.6 MB/s | 134 kB 00:00 2024-04-02T03:01:24.374 INFO:teuthology.orchestra.run.smithi186.stdout:(9/32): python3-cephfs-19.0.0-2461.gc2d32edc.el 2.7 MB/s | 159 kB 00:00 2024-04-02T03:01:24.425 INFO:teuthology.orchestra.run.smithi186.stdout:(10/32): python3-rados-19.0.0-2461.gc2d32edc.el 3.7 MB/s | 314 kB 00:00 2024-04-02T03:01:24.474 INFO:teuthology.orchestra.run.smithi186.stdout:(11/32): python3-rbd-19.0.0-2461.gc2d32edc.el9. 2.9 MB/s | 294 kB 00:00 2024-04-02T03:01:24.500 INFO:teuthology.orchestra.run.smithi186.stdout:(12/32): python3-rgw-19.0.0-2461.gc2d32edc.el9. 1.3 MB/s | 99 kB 00:00 2024-04-02T03:01:24.625 INFO:teuthology.orchestra.run.smithi186.stdout:(13/32): librgw2-19.0.0-2461.gc2d32edc.el9.x86_ 5.9 MB/s | 5.0 MB 00:00 2024-04-02T03:01:24.667 INFO:teuthology.orchestra.run.smithi186.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 547 kB/s | 106 kB 00:00 2024-04-02T03:01:24.701 INFO:teuthology.orchestra.run.smithi186.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 600 kB/s | 44 kB 00:00 2024-04-02T03:01:24.784 INFO:teuthology.orchestra.run.smithi186.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 617 kB/s | 159 kB 00:00 2024-04-02T03:01:24.968 INFO:teuthology.orchestra.run.smithi186.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 661 kB 00:00 2024-04-02T03:01:25.019 INFO:teuthology.orchestra.run.smithi186.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 890 kB/s | 282 kB 00:00 2024-04-02T03:01:25.044 INFO:teuthology.orchestra.run.smithi186.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 159 kB/s | 41 kB 00:00 2024-04-02T03:01:25.078 INFO:teuthology.orchestra.run.smithi186.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-04-02T03:01:25.103 INFO:teuthology.orchestra.run.smithi186.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 417 kB/s | 35 kB 00:00 2024-04-02T03:01:25.128 INFO:teuthology.orchestra.run.smithi186.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 554 kB/s | 46 kB 00:00 2024-04-02T03:01:25.253 INFO:teuthology.orchestra.run.smithi186.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-04-02T03:01:25.570 INFO:teuthology.orchestra.run.smithi186.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 153 kB/s | 49 kB 00:00 2024-04-02T03:01:25.812 INFO:teuthology.orchestra.run.smithi186.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 421 kB/s | 309 kB 00:00 2024-04-02T03:01:26.012 INFO:teuthology.orchestra.run.smithi186.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-04-02T03:01:26.046 INFO:teuthology.orchestra.run.smithi186.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 142 kB/s | 67 kB 00:00 2024-04-02T03:01:26.146 INFO:teuthology.orchestra.run.smithi186.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-02T03:01:26.296 INFO:teuthology.orchestra.run.smithi186.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2024-04-02T03:01:26.530 INFO:teuthology.orchestra.run.smithi186.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-04-02T03:01:26.705 INFO:teuthology.orchestra.run.smithi186.stdout:(31/32): librbd1-19.0.0-2461.gc2d32edc.el9.x86_ 9.0 MB/s | 3.7 MB 00:00 2024-04-02T03:01:26.764 INFO:teuthology.orchestra.run.smithi186.stdout:(32/32): librados2-19.0.0-2461.gc2d32edc.el9.x8 5.5 MB/s | 3.4 MB 00:00 2024-04-02T03:01:26.768 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:01:26.769 INFO:teuthology.orchestra.run.smithi186.stdout:Total 9.5 MB/s | 58 MB 00:06 2024-04-02T03:01:26.769 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:01:26.807 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:01:26.807 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:01:27.280 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:01:27.280 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:01:28.721 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:01:29.207 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-02T03:01:29.251 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/34 2024-04-02T03:01:29.355 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/34 2024-04-02T03:01:29.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-2461.gc2d32edc.el9.x86_64 3/34 2024-04-02T03:01:29.467 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-2461.gc2d32edc.el9.x86_64 4/34 2024-04-02T03:01:29.539 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-2461.gc2d32edc.el9.x86_64 4/34 2024-04-02T03:01:29.589 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-02T03:01:29.622 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-02T03:01:29.647 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-02T03:01:29.670 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-02T03:01:29.686 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-02T03:01:29.841 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-02T03:01:29.877 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-02T03:01:30.250 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-02T03:01:30.274 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-2461.gc2d32edc.el9.x86_64 11/34 2024-04-02T03:01:30.341 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-2461.gc2d32edc.el9.x86_64 11/34 2024-04-02T03:01:30.384 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-2461.gc2d32edc.el9. 12/34 2024-04-02T03:01:30.439 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-2461.gc2d32edc.el9.x86_64 13/34 2024-04-02T03:01:30.472 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 14/34 2024-04-02T03:01:30.560 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-02T03:01:30.594 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-2461.gc2d32edc.el9.x86_6 16/34 2024-04-02T03:01:30.682 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2461.gc2d32edc.el9.x86_6 16/34 2024-04-02T03:01:30.717 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-02T03:01:30.827 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-02T03:01:30.860 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-02T03:01:30.901 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-02T03:01:30.933 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-02T03:01:30.962 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-02T03:01:30.993 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-02T03:01:31.030 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-02T03:01:31.567 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-02T03:01:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-2461.gc2d32edc.el9.x86_64 26/34 2024-04-02T03:01:31.671 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-2461.gc2d32edc.el9.x86_64 26/34 2024-04-02T03:01:31.738 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 27/34 2024-04-02T03:01:31.769 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-2461.gc2d32edc.el9.x8 28/34 2024-04-02T03:01:34.062 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2461.gc2d32edc.el9.x86_64 29/34 2024-04-02T03:01:34.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-2461.gc2d32edc.el9.x86_64 29/34 2024-04-02T03:01:34.794 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2461.gc2d32edc.el9.x86_64 29/34 2024-04-02T03:01:34.827 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-2461.gc2d32edc.el9.x86_64 30/34 2024-04-02T03:01:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-2461.gc2d32edc.el9.x86_64 30/34 2024-04-02T03:01:34.915 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-02T03:01:34.915 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-02T03:01:34.915 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:34.953 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-2461.gc2d32edc.el9.x86_64 31/34 2024-04-02T03:01:44.970 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2461.gc2d32edc.el9.x86_64 31/34 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-02T03:01:44.971 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:45.971 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 32/34 2024-04-02T03:01:45.996 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 32/34 2024-04-02T03:01:45.996 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:01:45.996 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-02T03:01:45.996 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T03:01:45.996 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-02T03:01:45.996 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:45.998 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-02T03:01:46.090 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-02T03:01:46.091 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T03:01:47.360 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T03:01:47.360 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/34 2024-04-02T03:01:47.360 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 3/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-2461.gc2d32edc.el9.x86_64 4/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-2461.gc2d32edc.el9.x86_64 5/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-2461.gc2d32edc.el9.x86_6 6/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-2461.gc2d32edc.el9.x86_64 7/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2461.gc2d32edc.el9. 8/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-2461.gc2d32edc.el9.x8 9/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-2461.gc2d32edc.el9.x86_64 10/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-2461.gc2d32edc.el9.x86_64 11/34 2024-04-02T03:01:47.361 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 12/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 13/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-02T03:01:47.362 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-2461.gc2d32edc.el9.x86_64 31/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-02T03:01:47.363 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-2461.gc2d32edc.el9.x86_64 33/34 2024-04-02T03:01:48.382 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-02T03:01:48.382 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:48.382 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-02T03:01:48.382 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.382 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-02T03:01:48.383 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.384 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:48.385 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:01:48.700 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test 2024-04-02T03:01:49.264 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:31 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:01:49.427 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 87 M 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:01:49.428 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 173 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:01:49.429 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:01:49.430 INFO:teuthology.orchestra.run.smithi186.stdout:Install 11 Packages 2024-04-02T03:01:49.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:01:49.430 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 90 M 2024-04-02T03:01:49.430 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 819 M 2024-04-02T03:01:49.430 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:01:49.770 INFO:teuthology.orchestra.run.smithi186.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-04-02T03:01:49.845 INFO:teuthology.orchestra.run.smithi186.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-02T03:01:49.887 INFO:teuthology.orchestra.run.smithi186.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-04-02T03:01:49.929 INFO:teuthology.orchestra.run.smithi186.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-02T03:01:49.962 INFO:teuthology.orchestra.run.smithi186.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.7 MB/s | 90 kB 00:00 2024-04-02T03:01:50.096 INFO:teuthology.orchestra.run.smithi186.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-04-02T03:01:50.221 INFO:teuthology.orchestra.run.smithi186.stdout:(7/11): libcephsqlite-19.0.0-2461.gc2d32edc.el9 294 kB/s | 173 kB 00:00 2024-04-02T03:01:50.614 INFO:teuthology.orchestra.run.smithi186.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 477 kB/s | 187 kB 00:00 2024-04-02T03:01:50.639 INFO:teuthology.orchestra.run.smithi186.stdout:(9/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 864 kB/s | 469 kB 00:00 2024-04-02T03:01:50.922 INFO:teuthology.orchestra.run.smithi186.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 395 kB/s | 122 kB 00:00 2024-04-02T03:01:56.681 INFO:teuthology.orchestra.run.smithi186.stdout:(11/11): ceph-test-19.0.0-2461.gc2d32edc.el9.x8 12 MB/s | 87 MB 00:07 2024-04-02T03:01:56.685 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:01:56.685 INFO:teuthology.orchestra.run.smithi186.stdout:Total 12 MB/s | 90 MB 00:07 2024-04-02T03:01:56.726 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:01:56.749 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:01:56.749 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:01:57.405 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:01:57.405 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:01:58.430 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:01:58.501 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-02T03:01:58.540 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-02T03:01:58.566 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-02T03:01:58.635 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-02T03:01:58.665 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-02T03:01:58.710 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-02T03:01:58.755 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-02T03:01:58.795 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-02T03:01:58.834 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-02T03:01:58.881 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-02T03:01:58.907 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-2461.gc2d32edc.el9.x86_64 10/11 2024-04-02T03:02:10.564 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2461.gc2d32edc.el9.x86_64 10/11 2024-04-02T03:02:10.639 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-2461.gc2d32edc.el9.x86_64 11/11 2024-04-02T03:02:16.751 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-2:19.0.0-2461.gc2d32edc.el9.x86_64 11/11 2024-04-02T03:02:16.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/11 2024-04-02T03:02:16.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/11 2024-04-02T03:02:16.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-02T03:02:16.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-02T03:02:16.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-02T03:02:16.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-02T03:02:16.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-02T03:02:16.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-02T03:02:16.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-02T03:02:16.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-02T03:02:17.174 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-02T03:02:17.174 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-16.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:02:17.175 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:02:17.487 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph 2024-04-02T03:02:18.054 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:00 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:02:18.205 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 6.5 k 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 2.2 M 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 1.6 M 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 249 k 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 5.0 M 2024-04-02T03:02:18.208 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 71 M 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-02T03:02:18.209 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-02T03:02:18.210 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout:========================================================================================= 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout:Install 36 Packages 2024-04-02T03:02:18.211 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:02:18.212 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 84 M 2024-04-02T03:02:18.212 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 921 M 2024-04-02T03:02:18.212 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:02:19.041 INFO:teuthology.orchestra.run.smithi186.stdout:(1/36): ceph-19.0.0-2461.gc2d32edc.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2024-04-02T03:02:19.149 INFO:teuthology.orchestra.run.smithi186.stdout:(2/36): ceph-mgr-19.0.0-2461.gc2d32edc.el9.x86_ 4.1 MB/s | 1.6 MB 00:00 2024-04-02T03:02:19.983 INFO:teuthology.orchestra.run.smithi186.stdout:(3/36): ceph-mds-19.0.0-2461.gc2d32edc.el9.x86_ 1.8 MB/s | 2.2 MB 00:01 2024-04-02T03:02:20.150 INFO:teuthology.orchestra.run.smithi186.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2461.gc2d3 1.5 MB/s | 249 kB 00:00 2024-04-02T03:02:20.283 INFO:teuthology.orchestra.run.smithi186.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-04-02T03:02:20.325 INFO:teuthology.orchestra.run.smithi186.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-02T03:02:20.350 INFO:teuthology.orchestra.run.smithi186.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-04-02T03:02:20.392 INFO:teuthology.orchestra.run.smithi186.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-02T03:02:20.443 INFO:teuthology.orchestra.run.smithi186.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-04-02T03:02:20.510 INFO:teuthology.orchestra.run.smithi186.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-04-02T03:02:20.710 INFO:teuthology.orchestra.run.smithi186.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 5.7 MB/s | 1.1 MB 00:00 2024-04-02T03:02:20.752 INFO:teuthology.orchestra.run.smithi186.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-04-02T03:02:20.802 INFO:teuthology.orchestra.run.smithi186.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-02T03:02:20.853 INFO:teuthology.orchestra.run.smithi186.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-02T03:02:21.045 INFO:teuthology.orchestra.run.smithi186.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 651 kB/s | 125 kB 00:00 2024-04-02T03:02:21.320 INFO:teuthology.orchestra.run.smithi186.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-04-02T03:02:22.771 INFO:teuthology.orchestra.run.smithi186.stdout:(17/36): ceph-osd-19.0.0-2461.gc2d32edc.el9.x86 20 MB/s | 71 MB 00:03 2024-04-02T03:02:22.838 INFO:teuthology.orchestra.run.smithi186.stdout:(18/36): ceph-mon-19.0.0-2461.gc2d32edc.el9.x86 1.3 MB/s | 5.0 MB 00:03 2024-04-02T03:02:26.254 INFO:teuthology.orchestra.run.smithi186.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 50 kB/s | 172 kB 00:03 2024-04-02T03:02:26.429 INFO:teuthology.orchestra.run.smithi186.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-04-02T03:02:26.513 INFO:teuthology.orchestra.run.smithi186.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-04-02T03:02:26.618 INFO:teuthology.orchestra.run.smithi186.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 170 kB/s | 18 kB 00:00 2024-04-02T03:02:26.705 INFO:teuthology.orchestra.run.smithi186.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 264 kB/s | 23 kB 00:00 2024-04-02T03:02:26.789 INFO:teuthology.orchestra.run.smithi186.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-04-02T03:02:26.872 INFO:teuthology.orchestra.run.smithi186.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2024-04-02T03:02:26.956 INFO:teuthology.orchestra.run.smithi186.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-04-02T03:02:27.040 INFO:teuthology.orchestra.run.smithi186.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 947 kB/s | 79 kB 00:00 2024-04-02T03:02:27.148 INFO:teuthology.orchestra.run.smithi186.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-02T03:02:27.232 INFO:teuthology.orchestra.run.smithi186.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-02T03:02:27.315 INFO:teuthology.orchestra.run.smithi186.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-04-02T03:02:27.399 INFO:teuthology.orchestra.run.smithi186.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2024-04-02T03:02:27.499 INFO:teuthology.orchestra.run.smithi186.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-02T03:02:27.658 INFO:teuthology.orchestra.run.smithi186.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-04-02T03:02:27.758 INFO:teuthology.orchestra.run.smithi186.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-04-02T03:02:38.391 INFO:teuthology.orchestra.run.smithi186.stdout:(35/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.8 kB/s | 43 kB 00:15 2024-04-02T03:03:25.560 INFO:teuthology.orchestra.run.smithi186.stdout:(36/36): luarocks-3.9.2-1.el9.noarch.rpm 2.4 kB/s | 151 kB 01:04 2024-04-02T03:03:25.563 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:03:25.563 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.3 MB/s | 84 MB 01:07 2024-04-02T03:03:25.585 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:03:25.630 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:03:25.630 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:03:26.372 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:03:26.372 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:03:27.962 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:03:28.020 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-02T03:03:28.058 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-02T03:03:28.090 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-02T03:03:28.159 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-02T03:03:28.205 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-02T03:03:28.246 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-02T03:03:28.338 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-02T03:03:28.416 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-02T03:03:28.457 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-02T03:03:28.528 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-02T03:03:28.582 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-02T03:03:28.630 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-02T03:03:28.687 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-02T03:03:28.750 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-02T03:03:28.935 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-02T03:03:28.976 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-02T03:03:29.014 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-02T03:03:29.058 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-02T03:03:29.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-02T03:03:29.172 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-02T03:03:29.206 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-02T03:03:29.240 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-02T03:03:29.280 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-02T03:03:29.315 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-02T03:03:29.358 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-02T03:03:29.446 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-02T03:03:29.481 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-02T03:03:29.552 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-02T03:03:29.697 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-02T03:03:29.823 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-02T03:03:30.026 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2461.gc2d32edc.el9. 31/36 2024-04-02T03:03:30.053 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-2461.gc2d32edc.el9.x86_64 32/36 2024-04-02T03:03:30.078 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2461.gc2d32edc.el9.x86_64 32/36 2024-04-02T03:03:30.078 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:03:30.078 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-02T03:03:30.078 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T03:03:30.078 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-02T03:03:30.078 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:40.949 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-2461.gc2d32edc.el9.x86_64 33/36 2024-04-02T03:03:40.976 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-2461.gc2d32edc.el9.x86_64 33/36 2024-04-02T03:03:40.976 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:03:40.976 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-02T03:03:40.976 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T03:03:40.976 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-02T03:03:40.976 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:41.524 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-2461.gc2d32edc.el9.x86_64 34/36 2024-04-02T03:03:41.547 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-2461.gc2d32edc.el9.x86_64 34/36 2024-04-02T03:03:41.547 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:03:41.547 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-02T03:03:41.547 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T03:03:41.547 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-02T03:03:41.548 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:41.783 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-2461.gc2d32edc.el9.x86_64 35/36 2024-04-02T03:03:41.807 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-2461.gc2d32edc.el9.x86_64 35/36 2024-04-02T03:03:41.807 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:03:41.807 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-02T03:03:41.807 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T03:03:41.807 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-02T03:03:41.807 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:41.872 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-2461.gc2d32edc.el9.x86_64 36/36 2024-04-02T03:03:48.543 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-2:19.0.0-2461.gc2d32edc.el9.x86_64 36/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-2461.gc2d32edc.el9.x86_64 3/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-2461.gc2d32edc.el9.x86_64 4/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-2461.gc2d32edc.el9.x86_64 5/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2461.gc2d32edc.el9. 6/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-02T03:03:48.544 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-02T03:03:48.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-02T03:03:48.546 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-02T03:03:49.460 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-02T03:03:49.461 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-02T03:03:49.462 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-02T03:03:49.463 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-02T03:03:49.464 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-02T03:03:49.464 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-02T03:03:49.464 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-02T03:03:49.464 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-02T03:03:49.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:49.464 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:03:49.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base 2024-04-02T03:03:50.293 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:32 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:03:50.392 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:03:50.444 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:03:50.445 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:03:50.445 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:03:50.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install cephadm 2024-04-02T03:03:51.070 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:33 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:03:51.221 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 762 k 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:03:51.222 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:03:51.223 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:03:51.223 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:51.223 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 762 k 2024-04-02T03:03:51.223 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 768 k 2024-04-02T03:03:51.223 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:03:51.641 INFO:teuthology.orchestra.run.smithi186.stdout:cephadm-19.0.0-2461.gc2d32edc.el9.noarch.rpm 1.8 MB/s | 762 kB 00:00 2024-04-02T03:03:51.642 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:03:51.642 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.8 MB/s | 762 kB 00:00 2024-04-02T03:03:51.642 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:03:51.647 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:03:51.647 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:03:51.653 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:03:51.653 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:03:51.728 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:03:52.135 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 1/1 2024-04-02T03:03:52.238 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 1/1 2024-04-02T03:03:52.645 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 1/1 2024-04-02T03:03:52.944 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 1/1 2024-04-02T03:03:52.944 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:52.944 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:03:52.944 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:03:52.944 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:52.944 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:03:53.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache 2024-04-02T03:03:53.741 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:35 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:03:53.894 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:03:53.894 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:03:53.894 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-02T03:03:53.894 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 157 k 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 157 k 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 640 k 2024-04-02T03:03:53.895 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:03:54.273 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-19.0.0-2461.gc2d32e 416 kB/s | 157 kB 00:00 2024-04-02T03:03:54.274 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:03:54.274 INFO:teuthology.orchestra.run.smithi186.stdout:Total 415 kB/s | 157 kB 00:00 2024-04-02T03:03:54.274 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:03:54.283 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:03:54.283 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:03:54.335 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:03:54.336 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:03:54.477 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:03:54.557 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2461.gc2d32edc. 1/1 2024-04-02T03:03:54.581 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2461.gc2d32edc. 1/1 2024-04-02T03:03:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:03:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-02T03:03:54.581 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:55.545 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2461.gc2d32edc. 1/1 2024-04-02T03:03:55.545 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:55.545 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:03:55.545 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:03:55.545 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:55.545 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:03:55.777 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr 2024-04-02T03:03:56.357 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:38 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:03:56.457 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-mgr-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:03:56.509 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:03:56.510 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:03:56.510 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:03:56.570 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-dashboard 2024-04-02T03:03:57.132 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:39 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:03:57.285 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 3.5 M 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-02T03:03:57.286 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 34 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 16 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================== 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout:Install 11 Packages 2024-04-02T03:03:57.287 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:03:57.288 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 7.6 M 2024-04-02T03:03:57.288 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 100 M 2024-04-02T03:03:57.288 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:03:57.569 INFO:teuthology.orchestra.run.smithi186.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2461.gc2d 87 kB/s | 16 kB 00:00 2024-04-02T03:03:57.661 INFO:teuthology.orchestra.run.smithi186.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2461.gc2 121 kB/s | 34 kB 00:00 2024-04-02T03:03:57.695 INFO:teuthology.orchestra.run.smithi186.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-04-02T03:03:57.870 INFO:teuthology.orchestra.run.smithi186.stdout:(4/11): ceph-mgr-dashboard-19.0.0-2461.gc2d32ed 7.2 MB/s | 3.5 MB 00:00 2024-04-02T03:03:58.012 INFO:teuthology.orchestra.run.smithi186.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-04-02T03:03:58.079 INFO:teuthology.orchestra.run.smithi186.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-04-02T03:03:58.171 INFO:teuthology.orchestra.run.smithi186.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-04-02T03:03:58.221 INFO:teuthology.orchestra.run.smithi186.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 617 kB/s | 31 kB 00:00 2024-04-02T03:03:58.272 INFO:teuthology.orchestra.run.smithi186.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.9 MB/s | 2.0 MB 00:00 2024-04-02T03:03:58.313 INFO:teuthology.orchestra.run.smithi186.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-04-02T03:03:58.380 INFO:teuthology.orchestra.run.smithi186.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.2 MB/s | 863 kB 00:00 2024-04-02T03:03:58.385 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:03:58.385 INFO:teuthology.orchestra.run.smithi186.stdout:Total 6.9 MB/s | 7.6 MB 00:01 2024-04-02T03:03:58.402 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:03:58.420 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:03:58.420 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:03:58.598 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:03:58.598 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:03:59.088 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:03:59.186 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-02T03:03:59.234 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-02T03:03:59.346 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-02T03:03:59.484 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-02T03:03:59.537 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-02T03:03:59.629 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-02T03:03:59.680 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-02T03:03:59.710 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-02T03:03:59.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2461.gc2d32edc.el9 9/11 2024-04-02T03:04:00.802 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2461.gc2d32edc.el 10/11 2024-04-02T03:04:00.842 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2461.gc2d32edc.el9.noa 11/11 2024-04-02T03:04:02.195 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2461.gc2d32edc.el9.noa 11/11 2024-04-02T03:04:02.195 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2461.gc2d32edc.el 1/11 2024-04-02T03:04:02.195 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2461.gc2d32edc.el9.noa 2/11 2024-04-02T03:04:02.195 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2461.gc2d32edc.el9 3/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-02T03:04:02.196 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-02T03:04:02.558 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-02T03:04:02.559 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-02T03:04:02.559 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-02T03:04:02.559 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-02T03:04:02.559 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:02.559 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:04:02.804 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-02T03:04:03.367 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:45 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:04:03.519 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 7.4 M 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-02T03:04:03.520 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-04-02T03:04:03.521 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:03.522 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 37 M 2024-04-02T03:04:03.522 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 209 M 2024-04-02T03:04:03.522 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:04:04.006 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2024-04-02T03:04:04.039 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-04-02T03:04:04.090 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 696 kB/s | 34 kB 00:00 2024-04-02T03:04:04.148 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-04-02T03:04:04.357 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-04-02T03:04:04.424 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-02T03:04:04.449 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-04-02T03:04:04.824 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-02T03:04:05.591 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-24 4.3 MB/s | 7.4 MB 00:01 2024-04-02T03:04:05.742 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 219 kB/s | 195 kB 00:00 2024-04-02T03:04:05.825 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-04-02T03:04:06.309 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 432 kB/s | 803 kB 00:01 2024-04-02T03:04:06.312 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:04:06.312 INFO:teuthology.orchestra.run.smithi186.stdout:Total 13 MB/s | 37 MB 00:02 2024-04-02T03:04:06.352 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:04:06.377 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:04:06.378 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:04:06.728 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:04:06.728 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:04:07.441 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:04:07.516 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-02T03:04:07.555 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-02T03:04:07.677 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-02T03:04:07.725 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-02T03:04:07.904 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-02T03:04:07.941 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-02T03:04:08.020 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-02T03:04:08.681 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-02T03:04:08.804 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-02T03:04:10.070 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-02T03:04:11.099 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-02T03:04:11.141 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2461.gc2d32 12/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2461.gc2d32 12/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2461.gc2d32 1/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-02T03:04:13.707 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-02T03:04:13.708 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-02T03:04:13.708 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-02T03:04:13.708 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-02T03:04:13.708 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-02T03:04:14.144 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-02T03:04:14.145 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-02T03:04:14.146 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-02T03:04:14.146 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-02T03:04:14.146 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-02T03:04:14.146 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:14.146 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:04:14.568 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-rook 2024-04-02T03:04:15.143 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:57 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:04:15.296 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 49 k 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-02T03:04:15.297 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-02T03:04:15.298 INFO:teuthology.orchestra.run.smithi186.stdout:Install 13 Packages 2024-04-02T03:04:15.299 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:04:15.299 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.0 M 2024-04-02T03:04:15.299 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-04-02T03:04:15.299 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:04:15.501 INFO:teuthology.orchestra.run.smithi186.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 24 kB 00:00 2024-04-02T03:04:15.526 INFO:teuthology.orchestra.run.smithi186.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-04-02T03:04:15.576 INFO:teuthology.orchestra.run.smithi186.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-02T03:04:15.610 INFO:teuthology.orchestra.run.smithi186.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-04-02T03:04:15.635 INFO:teuthology.orchestra.run.smithi186.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-04-02T03:04:15.661 INFO:teuthology.orchestra.run.smithi186.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-04-02T03:04:15.686 INFO:teuthology.orchestra.run.smithi186.stdout:(7/13): ceph-mgr-rook-19.0.0-2461.gc2d32edc.el9 159 kB/s | 49 kB 00:00 2024-04-02T03:05:05.539 INFO:teuthology.orchestra.run.smithi186.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 660 B/s | 32 kB 00:49 2024-04-02T03:05:05.688 INFO:teuthology.orchestra.run.smithi186.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 6.8 MB/s | 1.0 MB 00:00 2024-04-02T03:05:05.730 INFO:teuthology.orchestra.run.smithi186.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-02T03:05:05.780 INFO:teuthology.orchestra.run.smithi186.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-04-02T03:05:08.440 INFO:teuthology.orchestra.run.smithi186.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 3.8 kB/s | 203 kB 00:52 2024-04-02T03:05:23.112 INFO:teuthology.orchestra.run.smithi186.stdout:(13/13): python3-certifi-2021.10.8-2.el9.noarch 227 B/s | 15 kB 01:07 2024-04-02T03:05:23.118 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:23.118 INFO:teuthology.orchestra.run.smithi186.stdout:Total 31 kB/s | 2.0 MB 01:07 2024-04-02T03:05:23.118 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:23.133 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:23.133 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:23.232 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:23.232 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:23.753 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:23.925 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-02T03:05:23.980 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-02T03:05:24.035 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-02T03:05:24.072 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-02T03:05:24.122 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-02T03:05:24.224 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-02T03:05:24.324 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-02T03:05:24.390 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-02T03:05:25.111 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-02T03:05:25.171 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-02T03:05:25.213 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-02T03:05:25.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-02T03:05:25.284 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-2461.gc2d32edc.el9.noarch 13/13 2024-04-02T03:05:26.040 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2461.gc2d32edc.el9.noarch 13/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2461.gc2d32edc.el9.noarch 1/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-02T03:05:26.041 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-02T03:05:26.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-02T03:05:26.042 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-02T03:05:26.405 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-02T03:05:26.405 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:26.405 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:26.405 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:05:26.405 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:26.406 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:26.645 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-cephadm 2024-04-02T03:05:27.214 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:09 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:27.365 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:27.366 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-02T03:05:27.366 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:05:27.366 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-02T03:05:27.366 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:27.366 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 141 k 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-04-02T03:05:27.367 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:27.368 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.8 M 2024-04-02T03:05:27.368 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 33 M 2024-04-02T03:05:27.368 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:27.790 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-02T03:05:27.991 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-04-02T03:05:28.018 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2461.gc2d32edc.e 351 kB/s | 141 kB 00:00 2024-04-02T03:05:28.119 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-04-02T03:05:28.144 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 378 kB/s | 58 kB 00:00 2024-04-02T03:05:28.169 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-04-02T03:05:28.171 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:28.171 INFO:teuthology.orchestra.run.smithi186.stdout:Total 8.5 MB/s | 6.8 MB 00:00 2024-04-02T03:05:28.185 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:28.195 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:28.195 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:28.285 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:28.286 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:28.619 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:28.777 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-02T03:05:28.829 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-02T03:05:29.356 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-02T03:05:29.431 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-02T03:05:29.494 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-02T03:05:29.533 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 6/6 2024-04-02T03:05:30.343 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 6/6 2024-04-02T03:05:30.344 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 1/6 2024-04-02T03:05:30.344 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-02T03:05:30.344 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-02T03:05:30.344 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-02T03:05:30.344 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-02T03:05:30.653 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-02T03:05:30.653 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:30.654 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:30.920 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse 2024-04-02T03:05:31.488 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:13 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:31.640 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 871 k 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:31.641 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 950 k 2024-04-02T03:05:31.642 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 4.0 M 2024-04-02T03:05:31.642 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:31.763 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 651 kB/s | 79 kB 00:00 2024-04-02T03:05:32.363 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-fuse-19.0.0-2461.gc2d32edc.el9.x86_ 1.2 MB/s | 871 kB 00:00 2024-04-02T03:05:32.364 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:32.365 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.3 MB/s | 950 kB 00:00 2024-04-02T03:05:32.365 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:32.387 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:32.388 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:32.461 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:32.461 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:32.670 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:32.822 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-02T03:05:32.889 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/2 2024-04-02T03:05:33.622 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/2 2024-04-02T03:05:33.622 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/2 2024-04-02T03:05:33.922 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-02T03:05:33.922 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:33.922 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:33.922 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-02T03:05:33.922 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:33.922 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:34.191 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-volume 2024-04-02T03:05:34.773 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:16 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:34.930 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:34.931 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:34.931 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:05:34.931 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:34.931 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:34.931 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-2461.gc2d32edc.el9 ceph-noarch 264 k 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 332 k 2024-04-02T03:05:34.933 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.5 M 2024-04-02T03:05:34.933 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:35.043 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 619 kB/s | 69 kB 00:00 2024-04-02T03:05:35.560 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-volume-19.0.0-2461.gc2d32edc.el9.no 420 kB/s | 264 kB 00:00 2024-04-02T03:05:35.561 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:35.562 INFO:teuthology.orchestra.run.smithi186.stdout:Total 527 kB/s | 332 kB 00:00 2024-04-02T03:05:35.562 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:35.572 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:35.572 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:35.615 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:35.615 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:35.843 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:36.024 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-02T03:05:36.071 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-2461.gc2d32edc.el9.noarch 2/2 2024-04-02T03:05:36.088 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-2461.gc2d32edc.el9.noarch 2/2 2024-04-02T03:05:36.088 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:05:36.088 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-02T03:05:36.088 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:36.816 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-2461.gc2d32edc.el9.noarch 1/2 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-2461.gc2d32edc.el9.noarch 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:37.156 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:37.454 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel 2024-04-02T03:05:38.037 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:20 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 125 k 2024-04-02T03:05:38.188 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 125 k 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 445 k 2024-04-02T03:05:38.189 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:38.575 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-19.0.0-2461.gc2d32edc.el9.x86_64 324 kB/s | 125 kB 00:00 2024-04-02T03:05:38.575 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:38.575 INFO:teuthology.orchestra.run.smithi186.stdout:Total 323 kB/s | 125 kB 00:00 2024-04-02T03:05:38.576 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:38.581 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:38.581 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:38.640 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:38.640 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:38.797 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:38.898 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:39.441 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:39.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:39.727 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:39.727 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:39.727 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:05:39.727 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:39.728 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:39.953 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2 2024-04-02T03:05:40.528 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:22 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:40.630 INFO:teuthology.orchestra.run.smithi186.stdout:Package libcephfs2-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:40.683 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:40.684 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:40.685 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:40.747 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs-devel 2024-04-02T03:05:41.312 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:23 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:41.462 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:41.463 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:41.463 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-02T03:05:41.463 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:41.463 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 32 k 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 32 k 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 142 k 2024-04-02T03:05:41.464 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:41.721 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs-devel-19.0.0-2461.gc2d32edc.el9.x86_6 124 kB/s | 32 kB 00:00 2024-04-02T03:05:41.721 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:41.721 INFO:teuthology.orchestra.run.smithi186.stdout:Total 123 kB/s | 32 kB 00:00 2024-04-02T03:05:41.722 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:41.725 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:41.725 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:41.757 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:41.758 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:41.845 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:41.969 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:42.390 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:42.642 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:42.642 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:42.642 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:42.642 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:05:42.642 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:42.642 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:42.809 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2 2024-04-02T03:05:43.371 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:25 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:43.469 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:43.522 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:43.523 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:43.523 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:43.591 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1 2024-04-02T03:05:44.156 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:26 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:44.255 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:44.307 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:44.308 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:44.309 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:44.374 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados 2024-04-02T03:05:44.939 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:26 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:45.038 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rados-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:45.090 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:45.091 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:45.091 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:45.155 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw 2024-04-02T03:05:45.720 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:27 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:45.819 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rgw-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:45.870 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:45.871 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:45.871 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:45.946 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs 2024-04-02T03:05:46.510 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:28 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:46.608 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-cephfs-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:46.659 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:46.660 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:46.661 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:46.728 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd 2024-04-02T03:05:47.296 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:29 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:47.395 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 is already installed. 2024-04-02T03:05:47.447 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:47.448 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-04-02T03:05:47.448 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:47.513 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse 2024-04-02T03:05:48.079 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:30 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:48.230 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:48.230 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 90 k 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 90 k 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 313 k 2024-04-02T03:05:48.231 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:48.449 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-19.0.0-2461.gc2d32edc.el9.x86_64.rpm 412 kB/s | 90 kB 00:00 2024-04-02T03:05:48.449 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:48.450 INFO:teuthology.orchestra.run.smithi186.stdout:Total 410 kB/s | 90 kB 00:00 2024-04-02T03:05:48.450 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:48.459 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:48.459 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:48.515 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:48.516 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:48.666 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:48.780 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:49.187 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:49.492 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:49.493 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:49.493 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:49.493 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:05:49.493 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:49.493 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:49.759 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror 2024-04-02T03:05:50.333 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:32 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:50.489 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:50.490 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:50.490 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-04-02T03:05:50.490 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 3.4 M 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 3.4 M 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 29 M 2024-04-02T03:05:50.491 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:52.553 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-19.0.0-2461.gc2d32edc.el9.x86_64.rpm 1.7 MB/s | 3.4 MB 00:02 2024-04-02T03:05:52.553 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:52.554 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.6 MB/s | 3.4 MB 00:02 2024-04-02T03:05:52.554 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:52.567 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:52.567 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:52.646 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:52.646 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:53.211 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:53.278 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:53.303 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:53.303 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-02T03:05:53.303 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-02T03:05:53.303 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T03:05:53.303 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-02T03:05:53.303 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:54.609 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:54.609 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:54.609 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:05:54.609 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:54.851 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd 2024-04-02T03:05:55.428 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:37 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:05:55.587 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 175 k 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:55.588 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 175 k 2024-04-02T03:05:55.589 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 673 k 2024-04-02T03:05:55.589 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:05:56.008 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-19.0.0-2461.gc2d32edc.el9.x86_64.rpm 419 kB/s | 175 kB 00:00 2024-04-02T03:05:56.008 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:05:56.008 INFO:teuthology.orchestra.run.smithi186.stdout:Total 417 kB/s | 175 kB 00:00 2024-04-02T03:05:56.009 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:05:56.018 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:05:56.018 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:05:56.078 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:05:56.078 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:05:56.271 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:05:56.392 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:56.893 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:57.191 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/1 2024-04-02T03:05:57.191 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:57.191 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:05:57.191 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:05:57.191 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:57.191 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:05:57.446 DEBUG:teuthology.parallel:result is None 2024-04-02T03:05:57.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=c2d32edc8c5fac8717d9c592e849ca877e2f524e 2024-04-02T03:05:57.677 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-02T03:05:57.698 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-2461.gc2d32edc.el9 2024-04-02T03:05:57.699 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2461.gc2d32edc.el9 2024-04-02T03:05:57.699 INFO:teuthology.task.install:The correct ceph version 19.0.0-2461.gc2d32edc is installed. 2024-04-02T03:05:57.700 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-02T03:05:57.700 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:57.701 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-02T03:05:57.767 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-02T03:05:57.768 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:57.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-02T03:05:57.835 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-02T03:05:57.903 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-02T03:05:57.903 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:57.903 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-02T03:05:57.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-02T03:05:58.036 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-02T03:05:58.036 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:58.037 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-02T03:05:58.126 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-02T03:05:58.193 INFO:teuthology.run_tasks:Running task ceph... 2024-04-02T03:05:58.277 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-02T03:05:58.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-04-02T03:05:58.305 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-02T03:05:58.305 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-02T03:05:58.375 INFO:tasks.ceph:Creating extra log directories... 2024-04-02T03:05:58.375 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-02T03:05:58.544 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-02T03:05:58.544 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-02T03:05:58.545 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285', 'branch': 'ci-yingxin-crimson-load-obc-3-base', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7635285', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-3-base'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-3-base', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-3-base', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c2d32edc8c5fac8717d9c592e849ca877e2f524e/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e', 'targets': {'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKCaDYUlgRjb+fcIIxGdyf1o5fBbkFNRF4sopkMh8EhVgc0uLzz6x1UIVrw/4ANIHPS2pef/KUQ70dpPvNVtOc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'c2d32edc8c5fac8717d9c592e849ca877e2f524e', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-04-02_02:43:05', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-02T03:05:58.545 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-02T03:05:58.563 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-04-02T03:05:58.630 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:58.630 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-02T03:05:58.688 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-02T03:05:58.689 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-02T03:05:58.743 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-02T03:05:58.743 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-02 03:05:53.560169836 +0000 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-02 03:00:33.546776243 +0000 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-02 03:00:33.546776243 +0000 2024-04-02T03:05:58.744 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-02 03:00:33.546776243 +0000 2024-04-02T03:05:58.744 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-02T03:05:58.810 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-02T03:05:58.810 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-02T03:05:58.810 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000183507 s, 2.8 MB/s 2024-04-02T03:05:58.811 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-02T03:05:58.866 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-02 03:05:53.560169836 +0000 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-02 03:00:33.898770004 +0000 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-02 03:00:33.898770004 +0000 2024-04-02T03:05:58.922 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-02 03:00:33.898770004 +0000 2024-04-02T03:05:58.923 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-02T03:05:58.987 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-02T03:05:58.987 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-02T03:05:58.987 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000199308 s, 2.6 MB/s 2024-04-02T03:05:58.988 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-02T03:05:59.043 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-02T03:05:59.097 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-02 03:05:53.561169819 +0000 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-02 03:00:34.234764049 +0000 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-02 03:00:34.234764049 +0000 2024-04-02T03:05:59.098 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-02 03:00:34.234764049 +0000 2024-04-02T03:05:59.098 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-02T03:05:59.163 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-02T03:05:59.163 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-02T03:05:59.163 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000191458 s, 2.7 MB/s 2024-04-02T03:05:59.165 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-02T03:05:59.220 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-02T03:05:59.275 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-02T03:05:59.275 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-02 03:05:53.561169819 +0000 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-02 03:00:34.580757917 +0000 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-02 03:00:34.580757917 +0000 2024-04-02T03:05:59.276 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-02 03:00:34.580757917 +0000 2024-04-02T03:05:59.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-02T03:05:59.341 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-02T03:05:59.341 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-02T03:05:59.342 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000225527 s, 2.3 MB/s 2024-04-02T03:05:59.342 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-02T03:05:59.398 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T03:05:59.398 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-02T03:05:59.399 INFO:tasks.ceph:Generating config... 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-02T03:05:59.400 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] log to file = False 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-02T03:05:59.401 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-02T03:05:59.401 INFO:tasks.ceph:[global] auth client required = none 2024-04-02T03:05:59.401 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-02T03:05:59.401 INFO:tasks.ceph:[global] auth service required = none 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] auth supported = none 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-02T03:05:59.402 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-02T03:05:59.403 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-02T03:05:59.404 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-02T03:05:59.405 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-02T03:05:59.406 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-02T03:05:59.407 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-02T03:05:59.407 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-02T03:05:59.407 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-02T03:05:59.407 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-02T03:05:59.407 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-02T03:05:59.407 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-02T03:05:59.407 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-02T03:05:59.407 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-02T03:05:59.407 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-02T03:05:59.407 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-02T03:05:59.407 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-02T03:05:59.407 INFO:tasks.ceph:Setting up mon.a... 2024-04-02T03:05:59.407 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-02T03:05:59.497 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.keyring 2024-04-02T03:05:59.500 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-02T03:05:59.553 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T03:05:59.579 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.186')] 2024-04-02T03:05:59.580 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.186', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-02T03:05:59.580 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:59.581 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T03:05:59.634 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.186 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:05:59.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:05:59.729+0000 7fadcf636f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:05:59.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:05:59.729+0000 7fadcf636f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:05:59.724 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:05:59.729+0000 7fadcf636f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:05:59.725 INFO:teuthology.orchestra.run.smithi186.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:05:59.725 INFO:teuthology.orchestra.run.smithi186.stdout:monmaptool: generated fsid 85f5d91c-b9b2-4998-b03e-6c2bd753750e 2024-04-02T03:05:59.725 INFO:teuthology.orchestra.run.smithi186.stdout:setting min_mon_release = quincy 2024-04-02T03:05:59.725 INFO:teuthology.orchestra.run.smithi186.stdout:epoch 0 2024-04-02T03:05:59.725 INFO:teuthology.orchestra.run.smithi186.stdout:fsid 85f5d91c-b9b2-4998-b03e-6c2bd753750e 2024-04-02T03:05:59.725 INFO:teuthology.orchestra.run.smithi186.stdout:last_changed 2024-04-02T03:05:59.730817+0000 2024-04-02T03:05:59.726 INFO:teuthology.orchestra.run.smithi186.stdout:created 2024-04-02T03:05:59.730817+0000 2024-04-02T03:05:59.726 INFO:teuthology.orchestra.run.smithi186.stdout:min_mon_release 17 (quincy) 2024-04-02T03:05:59.726 INFO:teuthology.orchestra.run.smithi186.stdout:election_strategy: 1 2024-04-02T03:05:59.726 INFO:teuthology.orchestra.run.smithi186.stdout:0: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.a 2024-04-02T03:05:59.726 INFO:teuthology.orchestra.run.smithi186.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-02T03:05:59.726 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-02T03:05:59.741 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 85f5d91c-b9b2-4998-b03e-6c2bd753750e... 2024-04-02T03:05:59.742 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-02T03:05:59.831 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-04-02T03:05:59.831 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-04-02T03:05:59.831 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = none 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-02T03:05:59.832 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-02T03:05:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = 172.21.15.186 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: auth client required = none 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: auth cluster required = none 2024-04-02T03:05:59.834 INFO:teuthology.orchestra.run.smithi186.stdout: auth service required = none 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug asok = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug bdev = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug buffer = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug civetweb = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug compressor = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug context = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug crush = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug crypto = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug dpdk = 0/0 2024-04-02T03:05:59.835 INFO:teuthology.orchestra.run.smithi186.stdout: debug eventtrace = 0/0 2024-04-02T03:05:59.836 INFO:teuthology.orchestra.run.smithi186.stdout: debug filer = 0/0 2024-04-02T03:05:59.836 INFO:teuthology.orchestra.run.smithi186.stdout: debug filestore = 0/0 2024-04-02T03:05:59.836 INFO:teuthology.orchestra.run.smithi186.stdout: debug finisher = 0/0 2024-04-02T03:05:59.836 INFO:teuthology.orchestra.run.smithi186.stdout: debug fuse = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug heartbeatmap = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug javaclient = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug journal = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug journaler = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug kstore = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug leveldb = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug lockdep = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds locker = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds log = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds log expire = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds migrator = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug memdb = 0/0 2024-04-02T03:05:59.837 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgrc = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug objclass = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug objectcacher = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug objecter = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug optracker = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug perfcounter = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug rados = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug rbd = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug rbd mirror = 0/0 2024-04-02T03:05:59.838 INFO:teuthology.orchestra.run.smithi186.stdout: debug rbd replay = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug refs = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug rgw = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug rgw lifecycle = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug rgw notification = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug rgw sync = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug striper = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug throttle = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug timer = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: debug tp = 0/0 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: ms client mode = crc 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: ms cluster mode = crc 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: ms mon client mode = crc 2024-04-02T03:05:59.839 INFO:teuthology.orchestra.run.smithi186.stdout: ms mon cluster mode = crc 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: ms mon service mode = crc 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: ms service mode = crc 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = 85f5d91c-b9b2-4998-b03e-6c2bd753750e 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.840 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-04-02T03:05:59.841 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore compression algorithm = snappy 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore compression mode = aggressive 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore rocksdb cf = False 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: crimson alien thread cpu cores = 6-7 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: crimson osd obc lru size = 10 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 20 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 20 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: debug filestore = 0/0 2024-04-02T03:05:59.842 INFO:teuthology.orchestra.run.smithi186.stdout: debug journal = 0/0 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 0/0 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 0/0 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 10 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: err to stderr = True 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: log flush on exit = True 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: log to file = False 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: log to stderr = True 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: osd shutdown pgref assert = True 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.843 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 0/0 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 0/0 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 0/0 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-04-02T03:05:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-04-02T03:05:59.845 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default crimson = True 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 0-2 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-04-02T03:05:59.846 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 3-5 2024-04-02T03:05:59.847 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-04-02T03:05:59.847 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 0-2 2024-04-02T03:05:59.847 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-04-02T03:05:59.847 INFO:teuthology.orchestra.run.smithi186.stdout: crimson seastar cpu cores = 3-5 2024-04-02T03:05:59.848 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-02T03:05:59.848 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-02T03:05:59.915 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-02T03:05:59.915 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:59.915 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-02T03:05:59.930 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:59.930 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-02T03:05:59.984 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:05:59.985 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:05:59.985 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-02T03:05:59.985 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-02T03:06:00.066 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:00.066 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:06:00.121 INFO:tasks.ceph:Setting up mon nodes... 2024-04-02T03:06:00.121 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-02T03:06:00.121 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T03:06:00.224 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-02T03:06:00.227 INFO:tasks.ceph:Setting up mds nodes... 2024-04-02T03:06:00.227 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-02T03:06:00.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-02T03:06:00.277 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-02T03:06:00.293 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-02T03:06:00.293 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-02T03:06:00.293 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-02T03:06:00.320 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T03:06:00.320 INFO:tasks.ceph:role: osd.0 2024-04-02T03:06:00.320 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:06:00.320 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T03:06:00.394 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T03:06:00.908 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-02T03:06:00.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-02T03:06:00.910 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-02T03:06:01.059 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-02T03:06:01.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-02T03:06:01.161 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T03:06:01.161 INFO:tasks.ceph:role: osd.1 2024-04-02T03:06:01.161 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:06:01.161 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T03:06:01.237 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T03:06:01.685 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-02T03:06:01.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-02T03:06:01.687 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-02T03:06:01.779 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-02T03:06:01.812 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-02T03:06:01.880 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-02T03:06:01.880 INFO:tasks.ceph:role: osd.2 2024-04-02T03:06:01.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:06:01.880 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-02T03:06:01.955 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-02T03:06:01.956 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-02T03:06:02.451 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-02T03:06:02.453 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-02T03:06:02.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-02T03:06:02.618 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-02T03:06:02.651 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:06:02.961 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T03:06:02.962 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-02 03:06:02,968 seastar - Reactor backend: linux-aio 2024-04-02T03:06:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:==29287==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T03:06:03.105 INFO:teuthology.orchestra.run.smithi186.stderr:DEBUG 2024-04-02 03:06:03,111 [shard 0:main] seastar - reactor::drain 2024-04-02T03:06:03.656 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:03.656 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-02T03:06:03.656 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-02T03:06:03.656 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-02T03:06:03.656 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:03.656 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:03.710 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T03:06:03.713 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-02 03:06:03,720 seastar - Reactor backend: linux-aio 2024-04-02T03:06:03.722 INFO:teuthology.orchestra.run.smithi186.stderr:==29285==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T03:06:08.378 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:08.378 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-02T03:06:08.378 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-02T03:06:08.378 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-02T03:06:08.379 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:08.379 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:08.379 INFO:teuthology.orchestra.run.smithi186.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 85f5d91c-b9b2-4998-b03e-6c2bd753750e 2024-04-02T03:06:08.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-02T03:06:08.429 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:06:08.717 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T03:06:09.582 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-02 03:06:08,724 seastar - Reactor backend: linux-aio 2024-04-02T03:06:09.582 INFO:teuthology.orchestra.run.smithi186.stderr:==29469==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:DEBUG 2024-04-02 03:06:08,863 [shard 0:main] seastar - reactor::drain 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-02 03:06:09,484 seastar - Reactor backend: linux-aio 2024-04-02T03:06:09.583 INFO:teuthology.orchestra.run.smithi186.stderr:==29467==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T03:06:14.141 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:14.141 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-02T03:06:14.141 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-02T03:06:14.142 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-02T03:06:14.142 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:14.142 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:14.142 INFO:teuthology.orchestra.run.smithi186.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 85f5d91c-b9b2-4998-b03e-6c2bd753750e 2024-04-02T03:06:14.163 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-02T03:06:14.193 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:06:14.487 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T03:06:14.488 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-02 03:06:14,494 seastar - Reactor backend: linux-aio 2024-04-02T03:06:14.493 INFO:teuthology.orchestra.run.smithi186.stderr:==29651==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T03:06:14.625 INFO:teuthology.orchestra.run.smithi186.stderr:DEBUG 2024-04-02 03:06:14,632 [shard 0:main] seastar - reactor::drain 2024-04-02T03:06:15.171 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:15.171 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-02T03:06:15.171 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-02T03:06:15.171 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-02T03:06:15.171 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:15.172 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:15.239 INFO:teuthology.orchestra.run.smithi186.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-02T03:06:15.243 INFO:teuthology.orchestra.run.smithi186.stderr:INFO 2024-04-02 03:06:15,250 seastar - Reactor backend: linux-aio 2024-04-02T03:06:15.252 INFO:teuthology.orchestra.run.smithi186.stderr:==29649==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-02T03:06:15.317 INFO:teuthology.orchestra.run.smithi186.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb99f799 0xb872260 0xb8737dc 0xb8739f2 0xb873b48 0xb874011 0x54daf 0x113bbf 0x4a0bd 0x35b7c 0x3629e 0x5db3a24 0x5db3c3a 0x5db3e1d 0x5db40f3 0x5e02933 0x5e02b39 0x5d81620 0x5d81b54 0x5bc6b33 0x5be1bf7 0x6ebc1b4 0x6ecd987 0x6ecee24 0x6eceea9 0x6ecefab 0x6ecf0a3 0x6ecf2b0 0x6ecf496 0x6ecf7e5 0x6ecfb89 0x6ecfdb9 0x6ed0023 0x6ed02f0 0x6ed050e 0x6ed05c6 0x6ed068c 0x6ed076d 0x5a2d13a 0x5a2d244 0x5a2d300 0x5a2d3bc 0x5a2d478 0x5a2d5c5 0x5a2d750 0x5a2db11 0x5a4a58e 0xb884ee5 0xb89f17f 0xb94017d 0xb9417cb 0xb61df33 0xb61e8af 0x368158a 0x3feaf 0x3ff5f 0x346d444 2024-04-02T03:06:15.317 INFO:teuthology.orchestra.run.smithi186.stderr:kernel callstack: 2024-04-02T03:06:19.923 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:Suppressions used: 2024-04-02T03:06:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: count bytes template 2024-04-02T03:06:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 1 8 MallocExtension::Register 2024-04-02T03:06:19.924 INFO:teuthology.orchestra.run.smithi186.stderr:----------------------------------------------------- 2024-04-02T03:06:19.924 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:19.924 INFO:teuthology.orchestra.run.smithi186.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 85f5d91c-b9b2-4998-b03e-6c2bd753750e 2024-04-02T03:06:19.946 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-02T03:06:19.975 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-02T03:06:19.975 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:19.975 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-02T03:06:20.042 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:20.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-02T03:06:20.108 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:20.109 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-02T03:06:20.178 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:20.178 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-02T03:06:20.241 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:20.241 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-02T03:06:20.297 INFO:tasks.ceph:Adding keys to all mons... 2024-04-02T03:06:20.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-02T03:06:20.362 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAYdgtmgrTKDRAAjQzeMkVdkmUyp5gePV8Ugw== 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAbdgtmbf9vNBAANTp/TOHj6hstVJ9uNOuZgw== 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAhdgtmvLZSJhAAUUPqn2g4nCGqzEnnlf366g== 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAndgtmX39UGBAA7tN1BYkHxVmuRvnJjbIssw== 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-04-02T03:06:20.363 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAYdgtmSz73EBAANaNyt7POmEC+GscBHcOZ7g== 2024-04-02T03:06:20.364 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-02T03:06:20.468 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T03:06:20.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T03:06:20.592 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-02T03:06:20.655 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-02T03:06:20.720 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-02T03:06:20.720 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-02T03:06:20.747 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-02T03:06:20.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:20.865+0000 7f736ae8da80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:20.859 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:20.865+0000 7f736ae8da80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:20.873 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-02T03:06:20.900 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-02T03:06:20.955 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-02T03:06:20.955 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-02T03:06:20.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-02T03:06:20.997 INFO:tasks.ceph.mon.a:Started 2024-04-02T03:06:20.997 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-02T03:06:20.997 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-02T03:06:20.998 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-02T03:06:21.000 INFO:tasks.ceph.mgr.x:Started 2024-04-02T03:06:21.001 INFO:tasks.ceph:Setting crush tunables to default 2024-04-02T03:06:21.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd crush tunables default 2024-04-02T03:06:21.084 INFO:tasks.ceph.mon.a.smithi186.stderr:2024-04-02T03:06:21.089+0000 7f719beb0a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.084 INFO:tasks.ceph.mon.a.smithi186.stderr:2024-04-02T03:06:21.089+0000 7f719beb0a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.084 INFO:tasks.ceph.mon.a.smithi186.stderr:2024-04-02T03:06:21.090+0000 7f719beb0a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.085 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:21.091+0000 7f4864d6b000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.114 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:21.115+0000 7f07bba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:21.122+0000 7f07bba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.185 INFO:teuthology.orchestra.run.smithi186.stderr:adjusted tunables profile to default 2024-04-02T03:06:21.199 INFO:tasks.ceph:check_enable_crimson: True 2024-04-02T03:06:21.200 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-02T03:06:21.200 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-02T03:06:21.289 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:21.295+0000 7f4864d6b000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.289 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:21.295+0000 7f4864d6b000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.309 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:21.311+0000 7f2f4282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.311 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:21.317+0000 7f2f4282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:21.391 INFO:tasks.ceph.mon.a.smithi186.stderr:2024-04-02T03:06:21.397+0000 7f7192bf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-02T03:06:21.391 INFO:tasks.ceph.mon.a.smithi186.stderr:Please be aware that this feature is experimental, untested, 2024-04-02T03:06:21.391 INFO:tasks.ceph.mon.a.smithi186.stderr:unsupported, and may result in data corruption, data loss, 2024-04-02T03:06:21.392 INFO:tasks.ceph.mon.a.smithi186.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-02T03:06:21.392 INFO:tasks.ceph.mon.a.smithi186.stderr:feature with important data. 2024-04-02T03:06:21.392 INFO:tasks.ceph.mon.a.smithi186.stderr: 2024-04-02T03:06:21.771 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:21.777+0000 7f4864d6b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-02T03:06:21.826 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:21.832+0000 7f4864d6b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-02T03:06:22.197 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-02T03:06:22.197 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:22.197 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-02T03:06:22.224 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:22.224 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-02T03:06:22.291 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:22.291 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-02T03:06:22.353 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.359+0000 7f4864d6b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-02T03:06:22.358 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new abeab761-28d1-4a0c-a905-d78b5fa9757a 0 2024-04-02T03:06:22.410 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.416+0000 7f4864d6b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-02T03:06:22.458 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.464+0000 7f4864d6b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-02T03:06:22.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:22.510+0000 7f4be88ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:22.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:22.517+0000 7f4be88ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:22.516 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.523+0000 7f4864d6b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-02T03:06:22.561 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.567+0000 7f4864d6b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-02T03:06:22.590 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:22.601 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 6c441d45-d1fe-4e45-bcb2-3b517c3c6cd7 1 2024-04-02T03:06:22.683 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.689+0000 7f4864d6b000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-02T03:06:22.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:22.753+0000 7f83b266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:22.754 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:22.760+0000 7f83b266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:22.758 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.764+0000 7f4864d6b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-02T03:06:22.807 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.813+0000 7f4864d6b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-02T03:06:22.832 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-04-02T03:06:22.844 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new f710a417-ebda-4f5a-bec1-29e644b97463 2 2024-04-02T03:06:22.862 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.868+0000 7f4864d6b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-02T03:06:22.912 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:22.918+0000 7f4864d6b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-02T03:06:22.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:22.973+0000 7fdf16f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:22.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:22.981+0000 7fdf16f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:23.053 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-04-02T03:06:23.064 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-02T03:06:23.096 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-02T03:06:23.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-02T03:06:23.139 INFO:tasks.ceph.osd.0:Started 2024-04-02T03:06:23.139 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-02T03:06:23.172 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-02T03:06:23.172 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-02T03:06:23.214 INFO:tasks.ceph.osd.1:Started 2024-04-02T03:06:23.215 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-02T03:06:23.249 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-02T03:06:23.249 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-02T03:06:23.291 INFO:tasks.ceph.osd.2:Started 2024-04-02T03:06:23.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T03:06:23.331 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:23.337+0000 7f4864d6b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-02T03:06:23.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:23.464+0000 7f86f046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:23.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:23.472+0000 7f86f046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:23.489 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:23.495+0000 7f4864d6b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-02T03:06:23.553 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:23.554 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":6,"fsid":"85f5d91c-b9b2-4998-b03e-6c2bd753750e","created":"2024-04-02T03:06:21.111109+0000","modified":"2024-04-02T03:06:23.059755+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T03:06:23.059755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"abeab761-28d1-4a0c-a905-d78b5fa9757a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c441d45-d1fe-4e45-bcb2-3b517c3c6cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f710a417-ebda-4f5a-bec1-29e644b97463","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T03:06:23.569 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-02T03:06:23.569 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-02T03:06:23.615 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:23.621+0000 7f4864d6b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-02T03:06:23.663 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:23.669+0000 7f4864d6b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-02T03:06:23.781 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:23.787+0000 7f4864d6b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-02T03:06:23.874 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T03:06:23.882 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:23.888+0000 7f4864d6b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-02T03:06:23.979 INFO:teuthology.misc.health.smithi186.stderr:2024-04-02T03:06:23.981+0000 7fc0b80ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:23.981 INFO:teuthology.misc.health.smithi186.stderr:2024-04-02T03:06:23.987+0000 7fc0b80ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:24.048 INFO:teuthology.misc.health.smithi186.stdout: 2024-04-02T03:06:24.048 INFO:teuthology.misc.health.smithi186.stdout:{"epoch":6,"fsid":"85f5d91c-b9b2-4998-b03e-6c2bd753750e","created":"2024-04-02T03:06:21.111109+0000","modified":"2024-04-02T03:06:23.059755+0000","last_up_change":"0.000000","last_in_change":"2024-04-02T03:06:23.059755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"abeab761-28d1-4a0c-a905-d78b5fa9757a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c441d45-d1fe-4e45-bcb2-3b517c3c6cd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f710a417-ebda-4f5a-bec1-29e644b97463","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T03:06:24.053 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:24.059+0000 7f4864d6b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-02T03:06:24.064 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-02T03:06:24.102 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:24.108+0000 7f4864d6b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-02T03:06:24.270 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:24.276+0000 7f4864d6b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-02T03:06:25.133 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:25.139+0000 7f4864d6b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-02T03:06:25.420 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:25.426+0000 7f4864d6b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-02T03:06:25.473 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:25.479+0000 7f4864d6b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-02T03:06:25.596 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:25.602+0000 7f4864d6b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-02T03:06:26.605 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-02T03:06:26.611+0000 7f4803f02640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-02T03:06:30.366 DEBUG:teuthology.orchestra.run.smithi186:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-02T03:06:30.478 INFO:teuthology.misc.health.smithi186.stderr:2024-04-02T03:06:30.481+0000 7f76abaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:30.480 INFO:teuthology.misc.health.smithi186.stderr:2024-04-02T03:06:30.487+0000 7f76abaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:30.677 INFO:teuthology.misc.health.smithi186.stdout: 2024-04-02T03:06:30.677 INFO:teuthology.misc.health.smithi186.stdout:{"epoch":9,"fsid":"85f5d91c-b9b2-4998-b03e-6c2bd753750e","created":"2024-04-02T03:06:21.111109+0000","modified":"2024-04-02T03:06:30.619681+0000","last_up_change":"2024-04-02T03:06:29.614669+0000","last_in_change":"2024-04-02T03:06:23.059755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T03:06:29.627541+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abeab761-28d1-4a0c-a905-d78b5fa9757a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":690918150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6805","nonce":690918150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6817","nonce":690918150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":690918150}]},"public_addr":"172.21.15.186:6804/690918150","cluster_addr":"172.21.15.186:6805/690918150","heartbeat_back_addr":"172.21.15.186:6817/690918150","heartbeat_front_addr":"172.21.15.186:6816/690918150","state":["exists","up"]},{"osd":1,"uuid":"6c441d45-d1fe-4e45-bcb2-3b517c3c6cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4144267598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":4144267598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4144267598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4144267598}]},"public_addr":"172.21.15.186:6808/4144267598","cluster_addr":"172.21.15.186:6809/4144267598","heartbeat_back_addr":"172.21.15.186:6810/4144267598","heartbeat_front_addr":"172.21.15.186:6807/4144267598","state":["exists","up"]},{"osd":2,"uuid":"f710a417-ebda-4f5a-bec1-29e644b97463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4005351681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6803","nonce":4005351681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6811","nonce":4005351681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4005351681}]},"public_addr":"172.21.15.186:6802/4005351681","cluster_addr":"172.21.15.186:6803/4005351681","heartbeat_back_addr":"172.21.15.186:6811/4005351681","heartbeat_front_addr":"172.21.15.186:6812/4005351681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T03:06:30.688 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-02T03:06:30.688 INFO:tasks.ceph:Creating RBD pool 2024-04-02T03:06:30.688 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-02T03:06:30.800 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:30.799+0000 7fe77ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:30.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:30.808+0000 7fe77ac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:31.620 INFO:teuthology.orchestra.run.smithi186.stderr:pool 'rbd' created 2024-04-02T03:06:31.643 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-02T03:06:31.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:31.774+0000 7f46a80ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:31.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:31.787+0000 7f46a80ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:32.624 INFO:teuthology.orchestra.run.smithi186.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-02T03:06:32.643 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-02T03:06:32.643 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-02T03:06:32.643 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-02T03:06:32.644 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-02T03:06:32.644 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-02T03:06:32.786 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:32.789+0000 7f8c6b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:32.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:32.794+0000 7f8c6b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.023 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:33.035 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3775444510},{"type":"v1","addr":"172.21.15.186:6801","nonce":3775444510}]},"active_addr":"172.21.15.186:6801/3775444510","active_change":"2024-04-02T03:06:25.604350+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3751361670}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3316351305}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3503655585}]}]} 2024-04-02T03:06:33.039 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-02T03:06:33.039 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-02T03:06:33.039 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T03:06:33.150 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.153+0000 7f8e3e660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.152 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.158+0000 7f8e3e660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.340 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:33.340 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":11,"fsid":"85f5d91c-b9b2-4998-b03e-6c2bd753750e","created":"2024-04-02T03:06:21.111109+0000","modified":"2024-04-02T03:06:32.628052+0000","last_up_change":"2024-04-02T03:06:29.614669+0000","last_in_change":"2024-04-02T03:06:23.059755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T03:06:29.627541+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-02T03:06:30.995921+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abeab761-28d1-4a0c-a905-d78b5fa9757a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":690918150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6805","nonce":690918150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6817","nonce":690918150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":690918150}]},"public_addr":"172.21.15.186:6804/690918150","cluster_addr":"172.21.15.186:6805/690918150","heartbeat_back_addr":"172.21.15.186:6817/690918150","heartbeat_front_addr":"172.21.15.186:6816/690918150","state":["exists","up"]},{"osd":1,"uuid":"6c441d45-d1fe-4e45-bcb2-3b517c3c6cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4144267598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":4144267598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4144267598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4144267598}]},"public_addr":"172.21.15.186:6808/4144267598","cluster_addr":"172.21.15.186:6809/4144267598","heartbeat_back_addr":"172.21.15.186:6810/4144267598","heartbeat_front_addr":"172.21.15.186:6807/4144267598","state":["exists","up"]},{"osd":2,"uuid":"f710a417-ebda-4f5a-bec1-29e644b97463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4005351681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6803","nonce":4005351681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6811","nonce":4005351681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4005351681}]},"public_addr":"172.21.15.186:6802/4005351681","cluster_addr":"172.21.15.186:6803/4005351681","heartbeat_back_addr":"172.21.15.186:6811/4005351681","heartbeat_front_addr":"172.21.15.186:6812/4005351681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T03:06:33.353 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-02T03:06:33.353 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-02T03:06:33.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.467+0000 7f43cb714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.472+0000 7f43cb714640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.659 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:33.659 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":12,"fsid":"85f5d91c-b9b2-4998-b03e-6c2bd753750e","created":"2024-04-02T03:06:21.111109+0000","modified":"2024-04-02T03:06:33.631253+0000","last_up_change":"2024-04-02T03:06:29.614669+0000","last_in_change":"2024-04-02T03:06:23.059755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-02T03:06:29.627541+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-02T03:06:30.995921+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abeab761-28d1-4a0c-a905-d78b5fa9757a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":690918150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6805","nonce":690918150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6817","nonce":690918150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":690918150}]},"public_addr":"172.21.15.186:6804/690918150","cluster_addr":"172.21.15.186:6805/690918150","heartbeat_back_addr":"172.21.15.186:6817/690918150","heartbeat_front_addr":"172.21.15.186:6816/690918150","state":["exists","up"]},{"osd":1,"uuid":"6c441d45-d1fe-4e45-bcb2-3b517c3c6cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4144267598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6809","nonce":4144267598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4144267598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4144267598}]},"public_addr":"172.21.15.186:6808/4144267598","cluster_addr":"172.21.15.186:6809/4144267598","heartbeat_back_addr":"172.21.15.186:6810/4144267598","heartbeat_front_addr":"172.21.15.186:6807/4144267598","state":["exists","up"]},{"osd":2,"uuid":"f710a417-ebda-4f5a-bec1-29e644b97463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4005351681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6803","nonce":4005351681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6811","nonce":4005351681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4005351681}]},"public_addr":"172.21.15.186:6802/4005351681","cluster_addr":"172.21.15.186:6803/4005351681","heartbeat_back_addr":"172.21.15.186:6811/4005351681","heartbeat_front_addr":"172.21.15.186:6812/4005351681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-02T03:06:33.672 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-02T03:06:33.672 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-02T03:06:33.672 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-02T03:06:33.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.792+0000 7fcf2442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.792+0000 7fdfde91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.792+0000 7f5f49b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.792 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.798+0000 7fdfde91d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.799+0000 7fcf2442d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.794 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.800+0000 7f5f49b14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.820 INFO:teuthology.orchestra.run.smithi186.stdout:34359738372 2024-04-02T03:06:33.820 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T03:06:33.824 INFO:teuthology.orchestra.run.smithi186.stdout:34359738372 2024-04-02T03:06:33.824 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T03:06:33.826 INFO:teuthology.orchestra.run.smithi186.stdout:34359738372 2024-04-02T03:06:33.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T03:06:33.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.938+0000 7f0e9bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.946+0000 7f0e9bb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.983 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.985+0000 7f4917f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.986 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.992+0000 7f4917f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:33.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:33.999+0000 7f344bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:34.000 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:34.006+0000 7f344bcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:34.159 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:34.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.1 2024-04-02T03:06:34.181 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:34.194 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.2 2024-04-02T03:06:34.198 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:34.211 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.0 2024-04-02T03:06:35.173 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T03:06:35.195 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T03:06:35.212 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T03:06:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:35.292+0000 7f22b6a60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:35.301+0000 7f22b6a60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:35.333 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:35.335+0000 7f6fea06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:35.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:35.335+0000 7f93b7dff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:35.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:35.342+0000 7f6fea06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:35.336 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:35.342+0000 7f93b6b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:35.493 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:35.506 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.1 2024-04-02T03:06:35.531 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:35.532 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:35.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.0 2024-04-02T03:06:35.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.2 2024-04-02T03:06:36.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T03:06:36.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T03:06:36.547 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T03:06:36.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:36.640+0000 7fd735b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:36.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:36.650+0000 7fd735b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:36.676 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:36.678+0000 7efd95e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:36.678 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:36.684+0000 7efd95e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:36.685 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:36.684+0000 7f58fcc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:36.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:36.695+0000 7f58fcc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:36.853 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:36.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.1 2024-04-02T03:06:36.876 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:36.887 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:36.889 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.2 2024-04-02T03:06:36.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.0 2024-04-02T03:06:37.867 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T03:06:37.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T03:06:37.901 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T03:06:37.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:37.997+0000 7fd6bddc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:38.001 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:38.008+0000 7fd6bddc6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:38.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:38.037+0000 7f84faf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:38.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:38.038+0000 7fe312e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:38.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:38.044+0000 7f84faf1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:38.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:38.044+0000 7fe312e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:38.215 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:38.228 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.1 2024-04-02T03:06:38.240 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:38.249 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:38.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.0 2024-04-02T03:06:38.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.2 2024-04-02T03:06:39.228 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T03:06:39.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T03:06:39.263 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T03:06:39.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:39.352+0000 7f49d955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:39.355 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:39.361+0000 7f49d955d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:39.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:39.393+0000 7f4352c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:39.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:39.400+0000 7f4352c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:39.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:39.396+0000 7ff8d34ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:39.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:39.403+0000 7ff8d34ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:39.575 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:39.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.1 2024-04-02T03:06:39.595 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:39.608 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-02T03:06:39.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.2 2024-04-02T03:06:39.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 0 for osd.0 2024-04-02T03:06:40.590 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-02T03:06:40.610 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-02T03:06:40.623 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-02T03:06:40.731 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:40.729+0000 7f7bfdc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:40.735 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:40.741+0000 7f7bfdc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:40.768 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:40.769+0000 7f029039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:40.769 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:40.771+0000 7fcdd546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:40.770 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:40.776+0000 7f029039d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:40.772 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:40.778+0000 7fcdd546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:40.960 INFO:teuthology.orchestra.run.smithi186.stdout:34359738378 2024-04-02T03:06:40.969 INFO:teuthology.orchestra.run.smithi186.stdout:34359738373 2024-04-02T03:06:40.973 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738378 for osd.1 2024-04-02T03:06:40.973 DEBUG:teuthology.parallel:result is None 2024-04-02T03:06:40.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.2 2024-04-02T03:06:40.983 DEBUG:teuthology.parallel:result is None 2024-04-02T03:06:40.999 INFO:teuthology.orchestra.run.smithi186.stdout:34359738373 2024-04-02T03:06:41.012 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738373 for osd.0 2024-04-02T03:06:41.012 DEBUG:teuthology.parallel:result is None 2024-04-02T03:06:41.012 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-02T03:06:41.013 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T03:06:41.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:41.169+0000 7fd18c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:41.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:41.174+0000 7fd18c75d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:41.349 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:41.349 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-04-02T03:06:41.361 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-02T03:06:39.639541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":421888,"data_stored":1251136,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.644277+0000","last_change":"2024-04-02T03:06:32.738206+0000","last_active":"2024-04-02T03:06:34.644277+0000","last_peered":"2024-04-02T03:06:34.644277+0000","last_clean":"2024-04-02T03:06:34.644277+0000","last_became_active":"2024-04-02T03:06:32.736320+0000","last_became_peered":"2024-04-02T03:06:32.736320+0000","last_unstale":"2024-04-02T03:06:34.644277+0000","last_undegraded":"2024-04-02T03:06:34.644277+0000","last_fullsized":"2024-04-02T03:06:34.644277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.643939+0000","last_change":"2024-04-02T03:06:32.743188+0000","last_active":"2024-04-02T03:06:34.643939+0000","last_peered":"2024-04-02T03:06:34.643939+0000","last_clean":"2024-04-02T03:06:34.643939+0000","last_became_active":"2024-04-02T03:06:32.740769+0000","last_became_peered":"2024-04-02T03:06:32.740769+0000","last_unstale":"2024-04-02T03:06:34.643939+0000","last_undegraded":"2024-04-02T03:06:34.643939+0000","last_fullsized":"2024-04-02T03:06:34.643939+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.642538+0000","last_change":"2024-04-02T03:06:32.722417+0000","last_active":"2024-04-02T03:06:34.642538+0000","last_peered":"2024-04-02T03:06:34.642538+0000","last_clean":"2024-04-02T03:06:34.642538+0000","last_became_active":"2024-04-02T03:06:32.718041+0000","last_became_peered":"2024-04-02T03:06:32.718041+0000","last_unstale":"2024-04-02T03:06:34.642538+0000","last_undegraded":"2024-04-02T03:06:34.642538+0000","last_fullsized":"2024-04-02T03:06:34.642538+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.644956+0000","last_change":"2024-04-02T03:06:32.753255+0000","last_active":"2024-04-02T03:06:34.644956+0000","last_peered":"2024-04-02T03:06:34.644956+0000","last_clean":"2024-04-02T03:06:34.644956+0000","last_became_active":"2024-04-02T03:06:32.750481+0000","last_became_peered":"2024-04-02T03:06:32.750481+0000","last_unstale":"2024-04-02T03:06:34.644956+0000","last_undegraded":"2024-04-02T03:06:34.644956+0000","last_fullsized":"2024-04-02T03:06:34.644956+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-02T03:06:32.728542+0000","last_change":"2024-04-02T03:06:32.728771+0000","last_active":"2024-04-02T03:06:32.728542+0000","last_peered":"2024-04-02T03:06:32.728542+0000","last_clean":"2024-04-02T03:06:31.624155+0000","last_became_active":"2024-04-02T03:06:32.713119+0000","last_became_peered":"2024-04-02T03:06:32.713119+0000","last_unstale":"2024-04-02T03:06:32.728542+0000","last_undegraded":"2024-04-02T03:06:32.728542+0000","last_fullsized":"2024-04-02T03:06:32.728542+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:33.658527+0000","last_change":"2024-04-02T03:06:32.686542+0000","last_active":"2024-04-02T03:06:33.658527+0000","last_peered":"2024-04-02T03:06:33.658527+0000","last_clean":"2024-04-02T03:06:33.658527+0000","last_became_active":"2024-04-02T03:06:32.684197+0000","last_became_peered":"2024-04-02T03:06:32.684197+0000","last_unstale":"2024-04-02T03:06:33.658527+0000","last_undegraded":"2024-04-02T03:06:33.658527+0000","last_fullsized":"2024-04-02T03:06:33.658527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:33.657420+0000","last_change":"2024-04-02T03:06:32.682752+0000","last_active":"2024-04-02T03:06:33.657420+0000","last_peered":"2024-04-02T03:06:33.657420+0000","last_clean":"2024-04-02T03:06:33.657420+0000","last_became_active":"2024-04-02T03:06:32.678673+0000","last_became_peered":"2024-04-02T03:06:32.678673+0000","last_unstale":"2024-04-02T03:06:33.657420+0000","last_undegraded":"2024-04-02T03:06:33.657420+0000","last_fullsized":"2024-04-02T03:06:33.657420+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.644681+0000","last_change":"2024-04-02T03:06:32.693365+0000","last_active":"2024-04-02T03:06:34.644681+0000","last_peered":"2024-04-02T03:06:34.644681+0000","last_clean":"2024-04-02T03:06:34.644681+0000","last_became_active":"2024-04-02T03:06:32.691025+0000","last_became_peered":"2024-04-02T03:06:32.691025+0000","last_unstale":"2024-04-02T03:06:34.644681+0000","last_undegraded":"2024-04-02T03:06:34.644681+0000","last_fullsized":"2024-04-02T03:06:34.644681+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.642228+0000","last_change":"2024-04-02T03:06:31.701227+0000","last_active":"2024-04-02T03:06:34.642228+0000","last_peered":"2024-04-02T03:06:34.642228+0000","last_clean":"2024-04-02T03:06:34.642228+0000","last_became_active":"2024-04-02T03:06:31.695972+0000","last_became_peered":"2024-04-02T03:06:31.695972+0000","last_unstale":"2024-04-02T03:06:34.642228+0000","last_undegraded":"2024-04-02T03:06:34.642228+0000","last_fullsized":"2024-04-02T03:06:34.642228+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:30.619681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:30.619681+0000","last_clean_scrub_stamp":"2024-04-02T03:06:30.619681+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26992,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344848,"statfs":{"total":96636764160,"available":96609124352,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":611906,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-02T03:06:41.362 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-02T03:06:41.474 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:41.477+0000 7f3262bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:41.476 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:41.482+0000 7f3262bf1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:41.664 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:41.665 INFO:teuthology.orchestra.run.smithi186.stderr:dumped all 2024-04-02T03:06:41.677 INFO:teuthology.orchestra.run.smithi186.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-02T03:06:41.608768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":420,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":430080,"data_stored":1254029,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.969226"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.644277+0000","last_change":"2024-04-02T03:06:32.738206+0000","last_active":"2024-04-02T03:06:34.644277+0000","last_peered":"2024-04-02T03:06:34.644277+0000","last_clean":"2024-04-02T03:06:34.644277+0000","last_became_active":"2024-04-02T03:06:32.736320+0000","last_became_peered":"2024-04-02T03:06:32.736320+0000","last_unstale":"2024-04-02T03:06:34.644277+0000","last_undegraded":"2024-04-02T03:06:34.644277+0000","last_fullsized":"2024-04-02T03:06:34.644277+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.643939+0000","last_change":"2024-04-02T03:06:32.743188+0000","last_active":"2024-04-02T03:06:34.643939+0000","last_peered":"2024-04-02T03:06:34.643939+0000","last_clean":"2024-04-02T03:06:34.643939+0000","last_became_active":"2024-04-02T03:06:32.740769+0000","last_became_peered":"2024-04-02T03:06:32.740769+0000","last_unstale":"2024-04-02T03:06:34.643939+0000","last_undegraded":"2024-04-02T03:06:34.643939+0000","last_fullsized":"2024-04-02T03:06:34.643939+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.642538+0000","last_change":"2024-04-02T03:06:32.722417+0000","last_active":"2024-04-02T03:06:34.642538+0000","last_peered":"2024-04-02T03:06:34.642538+0000","last_clean":"2024-04-02T03:06:34.642538+0000","last_became_active":"2024-04-02T03:06:32.718041+0000","last_became_peered":"2024-04-02T03:06:32.718041+0000","last_unstale":"2024-04-02T03:06:34.642538+0000","last_undegraded":"2024-04-02T03:06:34.642538+0000","last_fullsized":"2024-04-02T03:06:34.642538+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.644956+0000","last_change":"2024-04-02T03:06:32.753255+0000","last_active":"2024-04-02T03:06:34.644956+0000","last_peered":"2024-04-02T03:06:34.644956+0000","last_clean":"2024-04-02T03:06:34.644956+0000","last_became_active":"2024-04-02T03:06:32.750481+0000","last_became_peered":"2024-04-02T03:06:32.750481+0000","last_unstale":"2024-04-02T03:06:34.644956+0000","last_undegraded":"2024-04-02T03:06:34.644956+0000","last_fullsized":"2024-04-02T03:06:34.644956+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.670224+0000","last_change":"2024-04-02T03:06:32.728771+0000","last_active":"2024-04-02T03:06:34.670224+0000","last_peered":"2024-04-02T03:06:34.670224+0000","last_clean":"2024-04-02T03:06:34.670224+0000","last_became_active":"2024-04-02T03:06:32.713119+0000","last_became_peered":"2024-04-02T03:06:32.713119+0000","last_unstale":"2024-04-02T03:06:34.670224+0000","last_undegraded":"2024-04-02T03:06:34.670224+0000","last_fullsized":"2024-04-02T03:06:34.670224+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:33.658527+0000","last_change":"2024-04-02T03:06:32.686542+0000","last_active":"2024-04-02T03:06:33.658527+0000","last_peered":"2024-04-02T03:06:33.658527+0000","last_clean":"2024-04-02T03:06:33.658527+0000","last_became_active":"2024-04-02T03:06:32.684197+0000","last_became_peered":"2024-04-02T03:06:32.684197+0000","last_unstale":"2024-04-02T03:06:33.658527+0000","last_undegraded":"2024-04-02T03:06:33.658527+0000","last_fullsized":"2024-04-02T03:06:33.658527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:33.657420+0000","last_change":"2024-04-02T03:06:32.682752+0000","last_active":"2024-04-02T03:06:33.657420+0000","last_peered":"2024-04-02T03:06:33.657420+0000","last_clean":"2024-04-02T03:06:33.657420+0000","last_became_active":"2024-04-02T03:06:32.678673+0000","last_became_peered":"2024-04-02T03:06:32.678673+0000","last_unstale":"2024-04-02T03:06:33.657420+0000","last_undegraded":"2024-04-02T03:06:33.657420+0000","last_fullsized":"2024-04-02T03:06:33.657420+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.644681+0000","last_change":"2024-04-02T03:06:32.693365+0000","last_active":"2024-04-02T03:06:34.644681+0000","last_peered":"2024-04-02T03:06:34.644681+0000","last_clean":"2024-04-02T03:06:34.644681+0000","last_became_active":"2024-04-02T03:06:32.691025+0000","last_became_peered":"2024-04-02T03:06:32.691025+0000","last_unstale":"2024-04-02T03:06:34.644681+0000","last_undegraded":"2024-04-02T03:06:34.644681+0000","last_fullsized":"2024-04-02T03:06:34.644681+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:31.624155+0000","last_clean_scrub_stamp":"2024-04-02T03:06:31.624155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-02T03:06:34.642228+0000","last_change":"2024-04-02T03:06:31.701227+0000","last_active":"2024-04-02T03:06:34.642228+0000","last_peered":"2024-04-02T03:06:34.642228+0000","last_clean":"2024-04-02T03:06:34.642228+0000","last_became_active":"2024-04-02T03:06:31.695972+0000","last_became_peered":"2024-04-02T03:06:31.695972+0000","last_unstale":"2024-04-02T03:06:34.642228+0000","last_undegraded":"2024-04-02T03:06:34.642228+0000","last_fullsized":"2024-04-02T03:06:34.642228+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-02T03:06:30.619681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-02T03:06:30.619681+0000","last_clean_scrub_stamp":"2024-04-02T03:06:30.619681+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26992,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344848,"statfs":{"total":96636764160,"available":96609124352,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26992,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344848,"statfs":{"total":96636764160,"available":96609124352,"internally_reserved":0,"allocated":163840,"data_stored":614799,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-02T03:06:41.678 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-02T03:06:41.678 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-02T03:06:41.678 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-02T03:06:41.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-02T03:06:41.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:41.792+0000 7f087d114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:41.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:06:41.796+0000 7f087d114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:06:42.011 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:42.012 INFO:teuthology.orchestra.run.smithi186.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-02T03:06:42.024 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-02T03:06:42.025 INFO:teuthology.run_tasks:Running task cbt... 2024-04-02T03:06:42.043 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi186.front.sepia.ceph.com', 'osds': ['ubuntu@smithi186.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi186.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi186.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi186.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-02T03:06:42.043 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-02T03:06:42.062 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:06:42.062 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-02T03:06:42.115 DEBUG:teuthology.orchestra.run.smithi186:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-02T03:06:42.169 INFO:teuthology.orchestra.run.smithi186.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-02T03:06:42.509 INFO:tasks.cbt:Installing collectl 2024-04-02T03:06:42.509 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-02T03:06:42.610 INFO:teuthology.orchestra.run.smithi186.stderr:--2024-04-02 03:06:42-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-02T03:06:42.746 INFO:teuthology.orchestra.run.smithi186.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-04-02T03:06:42.755 INFO:teuthology.orchestra.run.smithi186.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-04-02T03:06:42.992 INFO:teuthology.orchestra.run.smithi186.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-02T03:06:42.992 INFO:teuthology.orchestra.run.smithi186.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmC3ZCmFoLzPX32V2e4t2yT0OSO99MepkBCpmJIEj4dlWE0Ag-iHBR9EiB0_B_QCGYAw7pKfhC8-JhBu_DoMj7iaX3ig%3D%3D&use_mirror=gigenet&r= [following] 2024-04-02T03:06:42.993 INFO:teuthology.orchestra.run.smithi186.stderr:--2024-04-02 03:06:42-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmC3ZCmFoLzPX32V2e4t2yT0OSO99MepkBCpmJIEj4dlWE0Ag-iHBR9EiB0_B_QCGYAw7pKfhC8-JhBu_DoMj7iaX3ig%3D%3D&use_mirror=gigenet&r= 2024-04-02T03:06:43.024 INFO:teuthology.orchestra.run.smithi186.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-02T03:06:43.105 INFO:teuthology.orchestra.run.smithi186.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-02T03:06:43.373 INFO:teuthology.orchestra.run.smithi186.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-02T03:06:43.373 INFO:teuthology.orchestra.run.smithi186.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-02T03:06:43.373 INFO:teuthology.orchestra.run.smithi186.stderr:--2024-04-02 03:06:43-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-02T03:06:43.386 INFO:teuthology.orchestra.run.smithi186.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2024-04-02T03:06:43.419 INFO:teuthology.orchestra.run.smithi186.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2024-04-02T03:06:43.603 INFO:teuthology.orchestra.run.smithi186.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-02T03:06:43.603 INFO:teuthology.orchestra.run.smithi186.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-02T03:06:43.603 INFO:teuthology.orchestra.run.smithi186.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-02T03:06:43.603 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:43.674 INFO:teuthology.orchestra.run.smithi186.stderr: 0K .......... .......... .......... .......... .......... 8% 701K 1s 2024-04-02T03:06:43.705 INFO:teuthology.orchestra.run.smithi186.stderr: 50K .......... .......... .......... .......... .......... 16% 1.55M 1s 2024-04-02T03:06:43.736 INFO:teuthology.orchestra.run.smithi186.stderr: 100K .......... .......... .......... .......... .......... 24% 1.60M 0s 2024-04-02T03:06:43.766 INFO:teuthology.orchestra.run.smithi186.stderr: 150K .......... .......... .......... .......... .......... 32% 1.61M 0s 2024-04-02T03:06:43.774 INFO:teuthology.orchestra.run.smithi186.stderr: 200K .......... .......... .......... .......... .......... 41% 5.98M 0s 2024-04-02T03:06:43.798 INFO:teuthology.orchestra.run.smithi186.stderr: 250K .......... .......... .......... .......... .......... 49% 2.07M 0s 2024-04-02T03:06:43.828 INFO:teuthology.orchestra.run.smithi186.stderr: 300K .......... .......... .......... .......... .......... 57% 1.61M 0s 2024-04-02T03:06:43.858 INFO:teuthology.orchestra.run.smithi186.stderr: 350K .......... .......... .......... .......... .......... 65% 1.63M 0s 2024-04-02T03:06:43.866 INFO:teuthology.orchestra.run.smithi186.stderr: 400K .......... .......... .......... .......... .......... 74% 5.78M 0s 2024-04-02T03:06:43.890 INFO:teuthology.orchestra.run.smithi186.stderr: 450K .......... .......... .......... .......... .......... 82% 2.07M 0s 2024-04-02T03:06:43.920 INFO:teuthology.orchestra.run.smithi186.stderr: 500K .......... .......... .......... .......... .......... 90% 1.66M 0s 2024-04-02T03:06:43.921 INFO:teuthology.orchestra.run.smithi186.stderr: 550K .......... .......... .......... .......... .......... 98% 31.4M 0s 2024-04-02T03:06:43.921 INFO:teuthology.orchestra.run.smithi186.stderr: 600K ...... 100% 11.4T=0.3s 2024-04-02T03:06:43.921 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:43.922 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02 03:06:43 (1.86 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-02T03:06:43.922 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:06:43.963 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/ 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/collectl.conf 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/ 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Socket.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/colmux.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Import.html 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/style.css 2024-04-02T03:06:43.964 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Process.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-02T03:06:43.965 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Export.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Startup.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Performance.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Messages.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Logging.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Misc.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Examples.html 2024-04-02T03:06:43.966 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Data.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Playback.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Hello.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Network.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-02T03:06:43.967 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Memory.html 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/Disks.html 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/service/ 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/service/collectl.service 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/initd/ 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/initd/collectl 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/misc.ph 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/gexpr.ph 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/statsd.ph 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/envrules.std 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/hello.ph 2024-04-02T03:06:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/vmsum.ph 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/proctree.ph 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/README-WINDOWS 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/formatit.ph 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/colmux 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/INSTALL 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/graphite.ph 2024-04-02T03:06:43.969 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-02T03:06:43.970 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/vmstat.ph 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/README 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/man1/ 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/man1/collectl.1 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/man1/colmux.1 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/lexpr.ph 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/client.pl 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/collectl 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/COPYING 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/vnet.ph 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/GPL 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/UNINSTALL 2024-04-02T03:06:43.971 INFO:teuthology.orchestra.run.smithi186.stdout:collectl-4.3.1/ARTISTIC 2024-04-02T03:06:44.020 INFO:teuthology.orchestra.run.smithi186.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-02T03:06:44.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-02T03:06:44.599 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:05:26 ago on Tue 02 Apr 2024 03:01:18 AM UTC. 2024-04-02T03:06:44.706 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-02T03:06:44.709 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-02T03:06:44.766 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-02T03:06:44.766 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:06:44.766 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: librbd-devel x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 23 k 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: perf x86_64 5.14.0-434.el9 CentOS-AppStream 2.2 M 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: libradospp-devel x86_64 2:19.0.0-2461.gc2d32edc.el9 ceph 30 k 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout:Install 5 Packages 2024-04-02T03:06:44.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:06:44.768 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.3 M 2024-04-02T03:06:44.768 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 11 M 2024-04-02T03:06:44.768 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-02T03:06:45.137 INFO:teuthology.orchestra.run.smithi186.stdout:(1/5): librbd-devel-19.0.0-2461.gc2d32edc.el9.x 76 kB/s | 23 kB 00:00 2024-04-02T03:06:45.182 INFO:teuthology.orchestra.run.smithi186.stdout:(2/5): perf-5.14.0-434.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-04-02T03:06:45.196 INFO:teuthology.orchestra.run.smithi186.stdout:(3/5): libradospp-devel-19.0.0-2461.gc2d32edc.e 84 kB/s | 30 kB 00:00 2024-04-02T03:06:45.780 INFO:teuthology.orchestra.run.smithi186.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 185 kB/s | 118 kB 00:00 2024-04-02T03:07:00.604 INFO:teuthology.orchestra.run.smithi186.stdout:(5/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 869 B/s | 13 kB 00:15 2024-04-02T03:07:00.605 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-02T03:07:00.606 INFO:teuthology.orchestra.run.smithi186.stdout:Total 151 kB/s | 2.3 MB 00:15 2024-04-02T03:07:00.606 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-02T03:07:00.629 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-02T03:07:00.629 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-02T03:07:00.761 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-02T03:07:00.762 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-02T03:07:01.043 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-02T03:07:01.114 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-02T03:07:01.161 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-02T03:07:01.192 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradospp-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 3/5 2024-04-02T03:07:01.500 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librbd-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 4/5 2024-04-02T03:07:01.581 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : perf-5.14.0-434.el9.x86_64 5/5 2024-04-02T03:07:02.315 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: perf-5.14.0-434.el9.x86_64 5/5 2024-04-02T03:07:02.315 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradospp-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 1/5 2024-04-02T03:07:02.315 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 2/5 2024-04-02T03:07:02.315 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : perf-5.14.0-434.el9.x86_64 3/5 2024-04-02T03:07:02.315 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-02T03:07:02.650 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-02T03:07:02.650 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:07:02.650 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-02T03:07:02.650 INFO:teuthology.orchestra.run.smithi186.stdout: libradospp-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:07:02.651 INFO:teuthology.orchestra.run.smithi186.stdout: librbd-devel-2:19.0.0-2461.gc2d32edc.el9.x86_64 2024-04-02T03:07:02.651 INFO:teuthology.orchestra.run.smithi186.stdout: pdsh-2.34-7.el9.x86_64 2024-04-02T03:07:02.651 INFO:teuthology.orchestra.run.smithi186.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-02T03:07:02.651 INFO:teuthology.orchestra.run.smithi186.stdout: perf-5.14.0-434.el9.x86_64 2024-04-02T03:07:02.651 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:07:02.651 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-02T03:07:03.026 INFO:tasks.cbt:benchmark: librbdfio 2024-04-02T03:07:03.026 DEBUG:teuthology.orchestra.run.smithi186:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-04-02T03:07:03.040 INFO:teuthology.orchestra.run.smithi186.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-04-02T03:07:05.208 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout:Operating system Linux 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout:CPU x86_64 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout:Big endian no 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout:Compiler gcc 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout:Cross compile no 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:07:05.775 INFO:teuthology.orchestra.run.smithi186.stdout:Static build no 2024-04-02T03:07:05.892 INFO:teuthology.orchestra.run.smithi186.stdout:Wordsize 64 2024-04-02T03:07:05.899 INFO:teuthology.orchestra.run.smithi186.stdout:zlib no 2024-04-02T03:07:05.958 INFO:teuthology.orchestra.run.smithi186.stdout:fcntl(F_FULLFSYNC) no 2024-04-02T03:07:05.972 INFO:teuthology.orchestra.run.smithi186.stdout:Linux AIO support no 2024-04-02T03:07:05.972 INFO:teuthology.orchestra.run.smithi186.stdout:Linux AIO support rw flags no 2024-04-02T03:07:06.032 INFO:teuthology.orchestra.run.smithi186.stdout:POSIX AIO support yes 2024-04-02T03:07:06.032 INFO:teuthology.orchestra.run.smithi186.stdout:POSIX AIO support needs -lrt no 2024-04-02T03:07:06.063 INFO:teuthology.orchestra.run.smithi186.stdout:POSIX AIO fsync yes 2024-04-02T03:07:06.094 INFO:teuthology.orchestra.run.smithi186.stdout:POSIX pshared support yes 2024-04-02T03:07:06.137 INFO:teuthology.orchestra.run.smithi186.stdout:pthread_condattr_setclock() yes 2024-04-02T03:07:06.178 INFO:teuthology.orchestra.run.smithi186.stdout:pthread_sigmask() yes 2024-04-02T03:07:06.215 INFO:teuthology.orchestra.run.smithi186.stdout:pthread_getaffinity_np() yes 2024-04-02T03:07:06.225 INFO:teuthology.orchestra.run.smithi186.stdout:Solaris AIO support no 2024-04-02T03:07:06.277 INFO:teuthology.orchestra.run.smithi186.stdout:__sync_fetch_and_add yes 2024-04-02T03:07:06.325 INFO:teuthology.orchestra.run.smithi186.stdout:__sync_synchronize yes 2024-04-02T03:07:06.354 INFO:teuthology.orchestra.run.smithi186.stdout:__sync_val_compare_and_swap yes 2024-04-02T03:07:06.362 INFO:teuthology.orchestra.run.smithi186.stdout:libverbs no 2024-04-02T03:07:06.386 INFO:teuthology.orchestra.run.smithi186.stdout:rdmacm no 2024-04-02T03:07:06.393 INFO:teuthology.orchestra.run.smithi186.stdout:librpma no 2024-04-02T03:07:06.403 INFO:teuthology.orchestra.run.smithi186.stdout:libprotobuf_c no 2024-04-02T03:07:06.434 INFO:teuthology.orchestra.run.smithi186.stdout:asprintf() yes 2024-04-02T03:07:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:vasprintf() yes 2024-04-02T03:07:06.496 INFO:teuthology.orchestra.run.smithi186.stdout:Linux fallocate yes 2024-04-02T03:07:06.528 INFO:teuthology.orchestra.run.smithi186.stdout:POSIX fadvise yes 2024-04-02T03:07:06.560 INFO:teuthology.orchestra.run.smithi186.stdout:POSIX fallocate yes 2024-04-02T03:07:06.590 INFO:teuthology.orchestra.run.smithi186.stdout:sched_setaffinity(3 arg) yes 2024-04-02T03:07:06.590 INFO:teuthology.orchestra.run.smithi186.stdout:sched_setaffinity(2 arg) no 2024-04-02T03:07:06.622 INFO:teuthology.orchestra.run.smithi186.stdout:clock_gettime yes 2024-04-02T03:07:06.655 INFO:teuthology.orchestra.run.smithi186.stdout:CLOCK_MONOTONIC yes 2024-04-02T03:07:06.702 INFO:teuthology.orchestra.run.smithi186.stdout:clockid_t yes 2024-04-02T03:07:06.737 INFO:teuthology.orchestra.run.smithi186.stdout:gettimeofday yes 2024-04-02T03:07:06.769 INFO:teuthology.orchestra.run.smithi186.stdout:fdatasync yes 2024-04-02T03:07:06.801 INFO:teuthology.orchestra.run.smithi186.stdout:pipe() yes 2024-04-02T03:07:06.832 INFO:teuthology.orchestra.run.smithi186.stdout:pipe2() yes 2024-04-02T03:07:06.862 INFO:teuthology.orchestra.run.smithi186.stdout:pread() yes 2024-04-02T03:07:06.947 INFO:teuthology.orchestra.run.smithi186.stdout:sync_file_range yes 2024-04-02T03:07:06.954 INFO:teuthology.orchestra.run.smithi186.stdout:ASharedMemory_create no 2024-04-02T03:07:06.986 INFO:teuthology.orchestra.run.smithi186.stdout:EXT4 move extent yes 2024-04-02T03:07:07.017 INFO:teuthology.orchestra.run.smithi186.stdout:Linux splice(2) yes 2024-04-02T03:07:07.024 INFO:teuthology.orchestra.run.smithi186.stdout:libnuma no 2024-04-02T03:07:07.054 INFO:teuthology.orchestra.run.smithi186.stdout:strsep yes 2024-04-02T03:07:07.086 INFO:teuthology.orchestra.run.smithi186.stdout:strcasestr yes 2024-04-02T03:07:07.098 INFO:teuthology.orchestra.run.smithi186.stdout:strlcat no 2024-04-02T03:07:07.131 INFO:teuthology.orchestra.run.smithi186.stdout:getopt_long_only() yes 2024-04-02T03:07:07.196 INFO:teuthology.orchestra.run.smithi186.stdout:inet_aton yes 2024-04-02T03:07:07.229 INFO:teuthology.orchestra.run.smithi186.stdout:socklen_t yes 2024-04-02T03:07:07.262 INFO:teuthology.orchestra.run.smithi186.stdout:__thread yes 2024-04-02T03:07:07.293 INFO:teuthology.orchestra.run.smithi186.stdout:RUSAGE_THREAD yes 2024-04-02T03:07:07.323 INFO:teuthology.orchestra.run.smithi186.stdout:SCHED_IDLE yes 2024-04-02T03:07:07.356 INFO:teuthology.orchestra.run.smithi186.stdout:TCP_NODELAY yes 2024-04-02T03:07:07.389 INFO:teuthology.orchestra.run.smithi186.stdout:vsock yes 2024-04-02T03:07:07.423 INFO:teuthology.orchestra.run.smithi186.stdout:Net engine window_size yes 2024-04-02T03:07:07.459 INFO:teuthology.orchestra.run.smithi186.stdout:TCP_MAXSEG yes 2024-04-02T03:07:07.488 INFO:teuthology.orchestra.run.smithi186.stdout:RLIMIT_MEMLOCK yes 2024-04-02T03:07:07.522 INFO:teuthology.orchestra.run.smithi186.stdout:pwritev/preadv yes 2024-04-02T03:07:07.555 INFO:teuthology.orchestra.run.smithi186.stdout:pwritev2/preadv2 yes 2024-04-02T03:07:07.603 INFO:teuthology.orchestra.run.smithi186.stdout:IPv6 helpers yes 2024-04-02T03:07:07.619 INFO:teuthology.orchestra.run.smithi186.stdout:http engine no 2024-04-02T03:07:07.763 INFO:teuthology.orchestra.run.smithi186.stdout:Rados engine yes 2024-04-02T03:07:07.920 INFO:teuthology.orchestra.run.smithi186.stdout:Rados Block Device engine yes 2024-04-02T03:07:08.076 INFO:teuthology.orchestra.run.smithi186.stdout:rbd_poll yes 2024-04-02T03:07:08.227 INFO:teuthology.orchestra.run.smithi186.stdout:rbd_invalidate_cache yes 2024-04-02T03:07:08.258 INFO:teuthology.orchestra.run.smithi186.stdout:setvbuf yes 2024-04-02T03:07:08.265 INFO:teuthology.orchestra.run.smithi186.stdout:Gluster API engine no 2024-04-02T03:07:08.281 INFO:teuthology.orchestra.run.smithi186.stdout:s390_z196_facilities no 2024-04-02T03:07:08.281 INFO:teuthology.orchestra.run.smithi186.stdout:HDFS engine no 2024-04-02T03:07:08.323 INFO:teuthology.orchestra.run.smithi186.stdout:MTD yes 2024-04-02T03:07:08.331 INFO:teuthology.orchestra.run.smithi186.stdout:libpmem no 2024-04-02T03:07:08.331 INFO:teuthology.orchestra.run.smithi186.stdout:libpmem1_5 no 2024-04-02T03:07:08.338 INFO:teuthology.orchestra.run.smithi186.stdout:libpmem2 no 2024-04-02T03:07:08.338 INFO:teuthology.orchestra.run.smithi186.stdout:PMDK dev-dax engine no 2024-04-02T03:07:08.338 INFO:teuthology.orchestra.run.smithi186.stdout:PMDK libpmem engine no 2024-04-02T03:07:08.345 INFO:teuthology.orchestra.run.smithi186.stdout:DDN's Infinite Memory Engine no 2024-04-02T03:07:08.345 INFO:teuthology.orchestra.run.smithi186.stdout:iscsi engine no 2024-04-02T03:07:08.345 INFO:teuthology.orchestra.run.smithi186.stdout:NBD engine no 2024-04-02T03:07:08.354 INFO:teuthology.orchestra.run.smithi186.stdout:DAOS File System (dfs) Engine no 2024-04-02T03:07:08.363 INFO:teuthology.orchestra.run.smithi186.stdout:NFS engine no 2024-04-02T03:07:08.363 INFO:teuthology.orchestra.run.smithi186.stdout:lex/yacc for arithmetic no 2024-04-02T03:07:08.402 INFO:teuthology.orchestra.run.smithi186.stdout:getmntent yes 2024-04-02T03:07:08.429 INFO:teuthology.orchestra.run.smithi186.stdout:getmntinfo no 2024-04-02T03:07:08.483 INFO:teuthology.orchestra.run.smithi186.stdout:Static Assert yes 2024-04-02T03:07:08.511 INFO:teuthology.orchestra.run.smithi186.stdout:bool yes 2024-04-02T03:07:08.544 INFO:teuthology.orchestra.run.smithi186.stdout:strndup yes 2024-04-02T03:07:08.552 INFO:teuthology.orchestra.run.smithi186.stdout:Valgrind headers no 2024-04-02T03:07:08.586 INFO:teuthology.orchestra.run.smithi186.stdout:Zoned block device support yes 2024-04-02T03:07:08.614 INFO:teuthology.orchestra.run.smithi186.stdout:Zoned block device capacity yes 2024-04-02T03:07:08.622 INFO:teuthology.orchestra.run.smithi186.stdout:libzbc engine no 2024-04-02T03:07:08.651 INFO:teuthology.orchestra.run.smithi186.stdout:NVMe uring command support yes 2024-04-02T03:07:08.666 INFO:teuthology.orchestra.run.smithi186.stdout:xnvme engine no 2024-04-02T03:07:08.674 INFO:teuthology.orchestra.run.smithi186.stdout:isal no 2024-04-02T03:07:08.688 INFO:teuthology.orchestra.run.smithi186.stdout:libblkio engine no 2024-04-02T03:07:08.688 INFO:teuthology.orchestra.run.smithi186.stdout:march_armv8_a_crc_crypto no 2024-04-02T03:07:08.689 INFO:teuthology.orchestra.run.smithi186.stdout:cuda no 2024-04-02T03:07:08.689 INFO:teuthology.orchestra.run.smithi186.stdout:libcufile no 2024-04-02T03:07:08.717 INFO:teuthology.orchestra.run.smithi186.stdout:Build march=native yes 2024-04-02T03:07:08.726 INFO:teuthology.orchestra.run.smithi186.stdout:CUnit no 2024-04-02T03:07:08.753 INFO:teuthology.orchestra.run.smithi186.stdout:__kernel_rwf_t yes 2024-04-02T03:07:08.781 INFO:teuthology.orchestra.run.smithi186.stdout:-Wimplicit-fallthrough=2 yes 2024-04-02T03:07:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:-Wno-stringop-truncation yes 2024-04-02T03:07:08.844 INFO:teuthology.orchestra.run.smithi186.stdout:MADV_HUGEPAGE yes 2024-04-02T03:07:08.875 INFO:teuthology.orchestra.run.smithi186.stdout:gettid yes 2024-04-02T03:07:08.909 INFO:teuthology.orchestra.run.smithi186.stdout:statx(2)/libc yes 2024-04-02T03:07:08.944 INFO:teuthology.orchestra.run.smithi186.stdout:statx(2)/syscall yes 2024-04-02T03:07:08.946 INFO:teuthology.orchestra.run.smithi186.stdout:Windows PDB generation no 2024-04-02T03:07:08.978 INFO:teuthology.orchestra.run.smithi186.stdout:timerfd_create yes 2024-04-02T03:07:08.979 INFO:teuthology.orchestra.run.smithi186.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-04-02T03:07:08.980 INFO:teuthology.orchestra.run.smithi186.stdout:Lib-based ioengines dynamic no 2024-04-02T03:07:09.043 INFO:teuthology.orchestra.run.smithi186.stdout:TCMalloc support yes 2024-04-02T03:07:09.044 INFO:teuthology.orchestra.run.smithi186.stdout:seed_buckets 4 2024-04-02T03:07:09.130 INFO:teuthology.orchestra.run.smithi186.stderr:FIO_VERSION = fio-3.37 2024-04-02T03:07:09.161 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc16.o 2024-04-02T03:07:09.239 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc32.o 2024-04-02T03:07:09.277 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc32c-arm64.o 2024-04-02T03:07:09.355 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc32c-intel.o 2024-04-02T03:07:09.399 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc32c.o 2024-04-02T03:07:09.439 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc64.o 2024-04-02T03:07:09.479 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crc7.o 2024-04-02T03:07:09.514 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/crct10dif_common.o 2024-04-02T03:07:09.549 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/fnv.o 2024-04-02T03:07:09.589 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/md5.o 2024-04-02T03:07:09.695 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/murmur3.o 2024-04-02T03:07:09.739 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/sha1.o 2024-04-02T03:07:09.928 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/sha256.o 2024-04-02T03:07:10.224 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/sha3.o 2024-04-02T03:07:10.368 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/sha512.o 2024-04-02T03:07:10.477 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/test.o 2024-04-02T03:07:10.645 INFO:teuthology.orchestra.run.smithi186.stdout: CC crc/xxhash.o 2024-04-02T03:07:10.742 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/axmap.o 2024-04-02T03:07:10.867 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/bloom.o 2024-04-02T03:07:10.950 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/flist_sort.o 2024-04-02T03:07:11.022 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/gauss.o 2024-04-02T03:07:11.112 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/getrusage.o 2024-04-02T03:07:11.148 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/hweight.o 2024-04-02T03:07:11.186 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/ieee754.o 2024-04-02T03:07:11.236 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/lfsr.o 2024-04-02T03:07:11.336 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/memalign.o 2024-04-02T03:07:11.381 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/memcpy.o 2024-04-02T03:07:11.523 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/mountcheck.o 2024-04-02T03:07:11.564 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/num2str.o 2024-04-02T03:07:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/output_buffer.o 2024-04-02T03:07:11.682 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/pattern.o 2024-04-02T03:07:11.827 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/prio_tree.o 2024-04-02T03:07:11.948 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/rand.o 2024-04-02T03:07:12.135 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/rbtree.o 2024-04-02T03:07:12.258 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/strntol.o 2024-04-02T03:07:12.305 INFO:teuthology.orchestra.run.smithi186.stdout: CC lib/zipf.o 2024-04-02T03:07:12.423 INFO:teuthology.orchestra.run.smithi186.stdout: CC gettime.o 2024-04-02T03:07:12.726 INFO:teuthology.orchestra.run.smithi186.stdout: CC ioengines.o 2024-04-02T03:07:13.005 INFO:teuthology.orchestra.run.smithi186.stdout: CC init.o 2024-04-02T03:07:13.780 INFO:teuthology.orchestra.run.smithi186.stdout: CC stat.o 2024-04-02T03:07:15.650 INFO:teuthology.orchestra.run.smithi186.stdout: CC log.o 2024-04-02T03:07:15.771 INFO:teuthology.orchestra.run.smithi186.stdout: CC time.o 2024-04-02T03:07:15.895 INFO:teuthology.orchestra.run.smithi186.stdout: CC filesetup.o 2024-04-02T03:07:16.599 INFO:teuthology.orchestra.run.smithi186.stdout: CC eta.o 2024-04-02T03:07:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: CC verify.o 2024-04-02T03:07:17.606 INFO:teuthology.orchestra.run.smithi186.stdout: CC memory.o 2024-04-02T03:07:17.781 INFO:teuthology.orchestra.run.smithi186.stdout: CC io_u.o 2024-04-02T03:07:18.566 INFO:teuthology.orchestra.run.smithi186.stdout: CC parse.o 2024-04-02T03:07:19.010 INFO:teuthology.orchestra.run.smithi186.stdout: CC fio_sem.o 2024-04-02T03:07:19.115 INFO:teuthology.orchestra.run.smithi186.stdout: CC rwlock.o 2024-04-02T03:07:19.200 INFO:teuthology.orchestra.run.smithi186.stdout: CC pshared.o 2024-04-02T03:07:19.267 INFO:teuthology.orchestra.run.smithi186.stdout: CC options.o 2024-04-02T03:07:20.057 INFO:teuthology.orchestra.run.smithi186.stdout: CC smalloc.o 2024-04-02T03:07:20.246 INFO:teuthology.orchestra.run.smithi186.stdout: CC filehash.o 2024-04-02T03:07:20.400 INFO:teuthology.orchestra.run.smithi186.stdout: CC profile.o 2024-04-02T03:07:20.519 INFO:teuthology.orchestra.run.smithi186.stdout: CC debug.o 2024-04-02T03:07:20.561 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/cpu.o 2024-04-02T03:07:20.707 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/mmap.o 2024-04-02T03:07:20.882 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/sync.o 2024-04-02T03:07:21.108 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/null.o 2024-04-02T03:07:21.227 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/net.o 2024-04-02T03:07:21.673 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/ftruncate.o 2024-04-02T03:07:21.770 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/fileoperations.o 2024-04-02T03:07:21.928 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/exec.o 2024-04-02T03:07:22.082 INFO:teuthology.orchestra.run.smithi186.stdout: CC server.o 2024-04-02T03:07:22.833 INFO:teuthology.orchestra.run.smithi186.stdout: CC client.o 2024-04-02T03:07:23.377 INFO:teuthology.orchestra.run.smithi186.stdout: CC iolog.o 2024-04-02T03:07:23.822 INFO:teuthology.orchestra.run.smithi186.stdout: CC backend.o 2024-04-02T03:07:24.670 INFO:teuthology.orchestra.run.smithi186.stdout: CC libfio.o 2024-04-02T03:07:24.861 INFO:teuthology.orchestra.run.smithi186.stdout: CC flow.o 2024-04-02T03:07:24.976 INFO:teuthology.orchestra.run.smithi186.stdout: CC cconv.o 2024-04-02T03:07:25.330 INFO:teuthology.orchestra.run.smithi186.stdout: CC gettime-thread.o 2024-04-02T03:07:25.436 INFO:teuthology.orchestra.run.smithi186.stdout: CC helpers.o 2024-04-02T03:07:25.496 INFO:teuthology.orchestra.run.smithi186.stdout: CC json.o 2024-04-02T03:07:25.681 INFO:teuthology.orchestra.run.smithi186.stdout: CC idletime.o 2024-04-02T03:07:25.908 INFO:teuthology.orchestra.run.smithi186.stdout: CC td_error.o 2024-04-02T03:07:26.007 INFO:teuthology.orchestra.run.smithi186.stdout: CC profiles/tiobench.o 2024-04-02T03:07:26.105 INFO:teuthology.orchestra.run.smithi186.stdout: CC profiles/act.o 2024-04-02T03:07:26.265 INFO:teuthology.orchestra.run.smithi186.stdout: CC io_u_queue.o 2024-04-02T03:07:26.315 INFO:teuthology.orchestra.run.smithi186.stdout: CC filelock.o 2024-04-02T03:07:26.442 INFO:teuthology.orchestra.run.smithi186.stdout: CC workqueue.o 2024-04-02T03:07:26.621 INFO:teuthology.orchestra.run.smithi186.stdout: CC rate-submit.o 2024-04-02T03:07:26.770 INFO:teuthology.orchestra.run.smithi186.stdout: CC optgroup.o 2024-04-02T03:07:26.816 INFO:teuthology.orchestra.run.smithi186.stdout: CC helper_thread.o 2024-04-02T03:07:26.993 INFO:teuthology.orchestra.run.smithi186.stdout: CC steadystate.o 2024-04-02T03:07:27.229 INFO:teuthology.orchestra.run.smithi186.stdout: CC zone-dist.o 2024-04-02T03:07:27.361 INFO:teuthology.orchestra.run.smithi186.stdout: CC zbd.o 2024-04-02T03:07:28.066 INFO:teuthology.orchestra.run.smithi186.stdout: CC dedupe.o 2024-04-02T03:07:28.212 INFO:teuthology.orchestra.run.smithi186.stdout: CC fdp.o 2024-04-02T03:07:28.350 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/posixaio.o 2024-04-02T03:07:28.486 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/falloc.o 2024-04-02T03:07:28.598 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/e4defrag.o 2024-04-02T03:07:28.742 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/splice.o 2024-04-02T03:07:28.936 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/asprintf.o 2024-04-02T03:07:28.971 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/strlcat.o 2024-04-02T03:07:29.015 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/mtd.o 2024-04-02T03:07:29.178 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/libmtd.o 2024-04-02T03:07:29.560 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/libmtd_legacy.o 2024-04-02T03:07:29.699 INFO:teuthology.orchestra.run.smithi186.stdout: CC diskutil.o 2024-04-02T03:07:29.909 INFO:teuthology.orchestra.run.smithi186.stdout: CC fifo.o 2024-04-02T03:07:29.965 INFO:teuthology.orchestra.run.smithi186.stdout: CC blktrace.o 2024-04-02T03:07:30.296 INFO:teuthology.orchestra.run.smithi186.stdout: CC cgroup.o 2024-04-02T03:07:30.449 INFO:teuthology.orchestra.run.smithi186.stdout: CC trim.o 2024-04-02T03:07:30.565 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/sg.o 2024-04-02T03:07:30.953 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/linux-dev-lookup.o 2024-04-02T03:07:31.011 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/io_uring.o 2024-04-02T03:07:31.396 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/nvme.o 2024-04-02T03:07:31.712 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/linux-blkzoned.o 2024-04-02T03:07:31.865 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/cmdprio.o 2024-04-02T03:07:32.108 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/rados.o 2024-04-02T03:07:32.281 INFO:teuthology.orchestra.run.smithi186.stdout: CC engines/rbd.o 2024-04-02T03:07:32.464 INFO:teuthology.orchestra.run.smithi186.stdout: CC fio.o 2024-04-02T03:07:32.553 INFO:teuthology.orchestra.run.smithi186.stdout: LINK fio 2024-04-02T03:07:32.810 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/log.o 2024-04-02T03:07:32.852 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/genzipf.o 2024-04-02T03:07:32.991 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/strcasestr.o 2024-04-02T03:07:33.014 INFO:teuthology.orchestra.run.smithi186.stdout: CC oslib/strndup.o 2024-04-02T03:07:33.037 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/fio-genzipf 2024-04-02T03:07:33.178 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/btrace2fio.o 2024-04-02T03:07:33.595 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/fio-btrace2fio 2024-04-02T03:07:33.735 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/verify-state.o 2024-04-02T03:07:33.832 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/debug.o 2024-04-02T03:07:33.868 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/fio-verify-state 2024-04-02T03:07:34.007 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/stest.o 2024-04-02T03:07:34.077 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/arch.o 2024-04-02T03:07:34.107 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/stest 2024-04-02T03:07:34.245 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/ieee754.o 2024-04-02T03:07:34.285 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/ieee754 2024-04-02T03:07:34.421 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/axmap.o 2024-04-02T03:07:34.522 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/axmap 2024-04-02T03:07:34.668 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/lfsr-test.o 2024-04-02T03:07:34.765 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/lfsr-test 2024-04-02T03:07:34.905 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/gen-rand.o 2024-04-02T03:07:35.003 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/gen-rand 2024-04-02T03:07:35.140 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/memlock.o 2024-04-02T03:07:35.204 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/memlock 2024-04-02T03:07:35.350 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/read-to-pipe-async.o 2024-04-02T03:07:35.539 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/read-to-pipe-async 2024-04-02T03:07:35.680 INFO:teuthology.orchestra.run.smithi186.stdout: CC t/io_uring.o 2024-04-02T03:07:36.220 INFO:teuthology.orchestra.run.smithi186.stdout: LINK t/io_uring 2024-04-02T03:07:36.359 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-02T03:07:36.388 DEBUG:teuthology.orchestra.run.smithi186:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - Settings.cluster: 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'clients': ['ubuntu@smithi186.front.sepia.ceph.com'], 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'head': 'ubuntu@smithi186.front.sepia.ceph.com', 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'is_teuthology': True, 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'iterations': 1, 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'mons': ['ubuntu@smithi186.front.sepia.ceph.com'], 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'osds': ['ubuntu@smithi186.front.sepia.ceph.com'], 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'osds_per_node': 3, 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'pid_dir': '/var/run/ceph', 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'rebuild_every_test': False, 2024-04-02T03:07:36.635 INFO:teuthology.orchestra.run.smithi186.stderr: 'rgws': {'ubuntu@smithi186.front.sepia.ceph.com': None}, 2024-04-02T03:07:36.636 INFO:teuthology.orchestra.run.smithi186.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-02T03:07:36.636 INFO:teuthology.orchestra.run.smithi186.stderr: 'use_existing': True, 2024-04-02T03:07:36.636 INFO:teuthology.orchestra.run.smithi186.stderr: 'user': 'ubuntu'} 2024-04-02T03:07:36.636 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.636 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.636 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.637 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi 2024-04-02T03:07:36.639 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.639 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.639 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-02T03:07:36.642 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-04-02T03:07:36.642 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.642 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.642 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl 2024-04-02T03:07:36.644 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.644 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T03:07:36.646 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.646 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.646 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/idle_monitoring/perf 2024-04-02T03:07:36.649 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:07:36.649 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T03:07:36.649 INFO:teuthology.orchestra.run.smithi186.stderr:03:07:36 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T03:08:36.698 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.699 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.699 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T03:08:36.738 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.738 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.738 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T03:08:36.774 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.774 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.774 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* 2024-04-02T03:08:36.791 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.791 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/$(basename ${i}).smithi186.front.sepia.ceph.com ; done 2024-04-02T03:08:36.796 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.796 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.796 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl 2024-04-02T03:08:36.798 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.798 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T03:08:36.800 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.800 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.800 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pool_monitoring/perf 2024-04-02T03:08:36.803 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.803 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T03:08:36.803 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T03:08:36.803 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.803 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:36.803 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-04-02T03:08:37.135 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:37 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:37.135 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:37 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:37.135 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-04-02T03:08:37.735 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:37 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:37.735 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:37 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:37.736 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-04-02T03:08:38.737 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:38 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:38.737 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:38 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:38.737 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-04-02T03:08:39.744 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:39.744 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:39 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:39.744 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-04-02T03:08:40.749 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:40 - INFO - cbt - Checking Health after pool creation. 2024-04-02T03:08:45.755 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:45 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:45.755 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:45 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:45.755 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-02T03:08:46.082 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:46 - INFO - cbt - Final Pool Health Check. 2024-04-02T03:08:51.088 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:51.088 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:51.088 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-04-02T03:08:51.428 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:51.428 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:51.428 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-04-02T03:08:51.430 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:51.430 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:51.431 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:51 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi186.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-04-02T03:08:53.017 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.017 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.018 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi186.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-04-02T03:08:53.131 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.131 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.131 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T03:08:53.150 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.151 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.151 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T03:08:53.183 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - INFO - cbt - Attempting to populating fio files... 2024-04-02T03:08:53.183 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.183 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.183 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-04-02T03:08:53.185 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.185 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:08:53.185 INFO:teuthology.orchestra.run.smithi186.stderr:03:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-04-02T03:11:03.870 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.873 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f. 2024-04-02T03:11:03.873 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - INFO - cbt - Setting OSD Read Ahead to: 4096 2024-04-02T03:11:03.873 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.873 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.873 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2024-04-02T03:11:03.874 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-02T03:11:03.874 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.874 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.874 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-02T03:11:03.887 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.887 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:03.887 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-04-02T03:11:04.998 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:04.998 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-04-02T03:11:05.745 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:05.745 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:05.745 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-02T03:11:05.865 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:05.865 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:05.865 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/ceph_settings.out 2024-04-02T03:11:31.899 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:31 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:31.900 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:31 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:31.900 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/pgautoscaler.log 2024-04-02T03:11:32.321 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.321 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.321 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/collectl 2024-04-02T03:11:32.324 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.324 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-04-02T03:11:32.325 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.325 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.325 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - PerfMonitoring: in local_node 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - INFO - cbt - Running rbd fio randrw test. 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.329 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.0 2024-04-02T03:11:32.330 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2024-04-02T03:11:32.330 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.330 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:11:32.330 INFO:teuthology.orchestra.run.smithi186.stderr:03:11:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randrw --output-format=json,normal --rwmixread=50 --rwmixwrite=50 --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/output.1 2024-04-02T03:12:35.204 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.205 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.205 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-04-02T03:12:35.229 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.230 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.230 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-04-02T03:12:35.259 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.259 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf.data 2024-04-02T03:12:35.260 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.260 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/perf/perf_stat.* 2024-04-02T03:12:35.262 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.262 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.262 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/historic_ops.out 2024-04-02T03:12:35.597 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.653 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.653 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* 2024-04-02T03:12:35.669 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - get_fqdn_local()=smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.669 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/$(basename ${i}).smithi186.front.sepia.ceph.com ; done 2024-04-02T03:12:35.704 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - INFO - cbt - Convert results to json format. 2024-04-02T03:12:35.704 INFO:teuthology.orchestra.run.smithi186.stderr:03:12:35 - DEBUG - cbt - Nodes : ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.704 INFO:teuthology.orchestra.run.smithi186.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-02T03:12:35.704 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com: smithi186.front.sepia.ceph.com 2024-04-02T03:12:35.704 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-02T03:12:35.704 INFO:teuthology.orchestra.run.smithi186.stdout:['smithi186.front.sepia.ceph.com'] 2024-04-02T03:12:35.732 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-02T03:12:35.742 DEBUG:teuthology.orchestra.run.smithi186:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-02T03:12:36.009 DEBUG:teuthology.orchestra.run.smithi186:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-04-02T03:12:36.128 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-02T03:12:36.128 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-02T03:12:36.128 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-02T03:12:36.177 INFO:teuthology.orchestra.run.smithi186.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.0.smithi186.front.sepia.ceph.com 2024-04-02T03:12:36.177 INFO:teuthology.orchestra.run.smithi186.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.1.smithi186.front.sepia.ceph.com 2024-04-02T03:12:36.178 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:12:36.178 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.0.smithi186.front.sepia.ceph.com of=/dev/stdout 2024-04-02T03:12:36.196 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T03:12:36.196 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f -name 'perf_stat.*' 2024-04-02T03:12:36.251 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T03:12:36.251 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T03:12:36.251 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-02T03:12:36.251 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f/json_output.1.smithi186.front.sepia.ceph.com of=/dev/stdout 2024-04-02T03:12:36.307 INFO:tasks.cbt_performance:reading total cpu cycles 2024-04-02T03:12:36.307 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-5928d42f -name 'perf_stat.*' 2024-04-02T03:12:36.362 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-04-02T03:12:36.362 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-04-02T03:12:36.374 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T03:12:36.383 INFO:tasks.cbt_performance:Data inserted successfully. 2024-04-02T03:12:36.383 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-02T03:12:36.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-02T03:12:36.516 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:12:36.515+0000 7fa0fb35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:12:36.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-02T03:12:36.524+0000 7fa0fb35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-02T03:12:36.732 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-02T03:12:36.732 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-02T03:12:36.733 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-02T03:12:36.733 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T03:12:41.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-02T03:12:42.834 INFO:tasks.ceph.osd.0:Stopped 2024-04-02T03:12:42.834 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-02T03:12:42.835 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T03:12:47.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-02T03:12:47.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-02T03:12:48.936 INFO:tasks.ceph.osd.1:Stopped 2024-04-02T03:12:48.936 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-02T03:12:48.936 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T03:12:52.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-02T03:12:52.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-02T03:12:55.037 INFO:tasks.ceph.osd.2:Stopped 2024-04-02T03:12:55.038 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-02T03:12:55.038 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-02T03:12:55.038 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T03:12:55.073 INFO:tasks.ceph.mgr.x:Stopped 2024-04-02T03:12:55.074 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-02T03:12:55.074 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-02T03:12:55.074 INFO:teuthology.orchestra.run:waiting for 300 2024-04-02T03:12:55.123 INFO:tasks.ceph.mon.a:Stopped 2024-04-02T03:12:55.124 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-02T03:12:55.124 DEBUG:teuthology.orchestra.run.smithi186:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-02T03:12:55.242 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:55.242 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-02T03:12:55.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:55.436 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-02T03:12:55.546 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:12:55.546 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-02T03:12:55.604 INFO:tasks.ceph:Archiving mon data... 2024-04-02T03:12:55.605 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285/data/mon.a.tgz 2024-04-02T03:12:55.606 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-02T03:12:55.622 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.Eop9AeFvm3 2024-04-02T03:12:55.623 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Eop9AeFvm3 2024-04-02T03:12:55.753 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.Eop9AeFvm3 2024-04-02T03:12:55.869 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.Eop9AeFvm3 is 204KB 2024-04-02T03:12:55.922 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.Eop9AeFvm3 2024-04-02T03:12:55.937 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-02T03:12:55.937 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-02T03:12:56.075 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-02T03:12:56.075 INFO:tasks.ceph:Archiving crash dumps... 2024-04-02T03:12:56.077 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285/remote/smithi186/crash 2024-04-02T03:12:56.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-02T03:12:56.141 INFO:tasks.ceph:Compressing logs... 2024-04-02T03:12:56.142 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-02T03:12:56.231 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30450.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31874.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32254.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30450.log: /var/log/ceph/ceph-client.admin.31874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31369.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32254.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31708.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35682.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30410.log 2024-04-02T03:12:56.232 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32086.log 2024-04-02T03:12:56.238 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.30410.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35706.log 2024-04-02T03:12:56.238 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35754.log 2024-04-02T03:12:56.238 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-04-02T03:12:56.238 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31960.log 2024-04-02T03:12:56.238 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31496.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31960.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35723.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35559.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/ceph-client.admin.35723.log: --verbose -- /var/log/ceph/ceph-client.admin.30370.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.35559.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35586.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35983.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31621.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35495.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35983.log.gz 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31101.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35495.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31370.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31747.log 2024-04-02T03:12:56.239 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.31370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35984.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31748.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30638.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35984.log.gz 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.31748.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31246.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35611.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.31246.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32212.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32296.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30172.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31622.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32296.log: /var/log/ceph/ceph-client.admin.30172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31999.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31204.log 2024-04-02T03:12:56.240 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31622.log: /var/log/ceph/ceph-client.admin.31999.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-02T03:12:56.245 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32126.log 2024-04-02T03:12:56.245 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31204.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31495.log 2024-04-02T03:12:56.245 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31288.log 2024-04-02T03:12:56.245 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32126.log: /var/log/ceph/ceph-client.admin.31495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31143.log 2024-04-02T03:12:56.245 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35636.log/var/log/ceph/ceph-client.admin.31288.log: 2024-04-02T03:12:56.245 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31143.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28813.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31834.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35636.log: /var/log/ceph/ceph.tmp-client.admin.28813.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36271.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35659.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.36271.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36271.log.gz 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.35659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31045.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32125.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31045.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32000.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35755.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31582.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31368.log 2024-04-02T03:12:56.246 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35936.log 2024-04-02T03:12:56.247 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30260.log 2024-04-02T03:12:56.247 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31456.log 2024-04-02T03:12:56.247 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31873.log 2024-04-02T03:12:56.309 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32125.log: /var/log/ceph/ceph-client.admin.35755.log: /var/log/ceph/ceph-client.admin.31873.log: /var/log/ceph/ceph-client.admin.31456.log: /var/log/ceph/ceph-client.admin.31368.log: /var/log/ceph/ceph-client.admin.31582.log: /var/log/ceph/ceph-client.admin.32000.log: /var/log/ceph/ceph-client.admin.30260.log: /var/log/ceph/ceph-client.admin.35936.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35559.log.gz 2024-04-02T03:12:56.309 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35495.log.gz 2024-04-02T03:12:56.309 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35586.log.gz 2024-04-02T03:12:56.309 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35611.log.gz 2024-04-02T03:12:56.309 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35936.log.gz 2024-04-02T03:12:56.309 INFO:teuthology.orchestra.run.smithi186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35636.log.gz 2024-04-02T03:12:56.337 INFO:teuthology.orchestra.run.smithi186.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-02T03:12:56.337 INFO:teuthology.orchestra.run.smithi186.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28813.log.gz 2024-04-02T03:12:56.339 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31045.log.gz 2024-04-02T03:12:56.339 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31101.log.gz 2024-04-02T03:12:56.339 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31143.log.gz 2024-04-02T03:12:56.340 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31204.log.gz 2024-04-02T03:12:56.340 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31246.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31288.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31368.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31369.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31370.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31456.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31495.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31496.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31582.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31621.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.31622.log.gz 2024-04-02T03:12:56.343 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31748.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31747.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31708.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31834.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31873.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31874.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31960.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31999.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32000.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32086.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32125.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32126.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32254.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32212.log.gz 2024-04-02T03:12:56.344 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32296.log.gz 2024-04-02T03:12:56.376 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35659.log.gz 2024-04-02T03:12:56.376 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35682.log.gz 65.3% 2024-04-02T03:12:56.376 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35706.log.gz 2024-04-02T03:12:56.376 INFO:teuthology.orchestra.run.smithi186.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.35723.log.gz 2024-04-02T03:12:56.377 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35755.log.gz 2024-04-02T03:12:56.377 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35754.log.gz 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% 50.4% 50.4% 86.3% -- replaced with /var/log/ceph/ceph-client.admin.30410.log.gz 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30172.log.gz -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 50.0% 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: 87.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz -- replaced with /var/log/ceph/ceph-client.admin.30450.log.gz 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-02T03:12:56.389 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30370.log.gz 2024-04-02T03:12:56.390 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30260.log.gz 2024-04-02T03:12:56.390 INFO:teuthology.orchestra.run.smithi186.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30638.log.gz 2024-04-02T03:12:57.286 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-02T03:13:14.311 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-02T03:13:14.431 INFO:teuthology.orchestra.run.smithi186.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-02T03:13:14.685 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-02T03:13:14.687 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-02T03:13:14.687 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m18.491s 2024-04-02T03:13:14.687 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m12.468s 2024-04-02T03:13:14.687 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.664s 2024-04-02T03:13:14.687 INFO:tasks.ceph:Archiving logs... 2024-04-02T03:13:14.688 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285/remote/smithi186/log 2024-04-02T03:13:14.688 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-02T03:13:16.561 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-02T03:13:16.572 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-02T03:13:16.572 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-02T03:13:16.651 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-02T03:13:16.660 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-02T03:13:16.660 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-02T03:13:16.676 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-02T03:13:16.752 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-02T03:13:16.752 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-02T03:13:16.752 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 -980us[ -980us] +/- 100ms 2024-04-02T03:13:16.752 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -373us[ -372us] +/- 68ms 2024-04-02T03:13:16.753 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +1118us[+1119us] +/- 78ms 2024-04-02T03:13:16.753 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-02T03:13:16.765 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-02T03:13:16.774 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-02T03:13:16.775 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-02T03:13:16.783 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-02T03:13:16.925 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-02T03:13:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T03:13:16.954 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-02T03:13:16.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-02T03:13:16.973 INFO:teuthology.task.internal:Duration was 997.352562 seconds 2024-04-02T03:13:16.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-02T03:13:16.982 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-02T03:13:16.982 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-02T03:13:17.101 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-02T03:13:17.209 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-02T03:13:17.209 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-02T03:13:17.209 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-02T03:13:17.228 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-02T03:13:17.228 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-02T03:13:17.297 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-02T03:13:17.298 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-02T03:13:17.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-02T03:13:17.449 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-02T03:13:17.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-02T03:13:17.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-02T03:13:17.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-02T03:13:17.558 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-02T03:13:17.578 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-02T03:13:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-02T03:13:17.633 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-02T03:13:17.644 INFO:teuthology.task.internal:Transferring archived files... 2024-04-02T03:13:17.644 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635285/remote/smithi186 2024-04-02T03:13:17.645 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-02T03:13:17.743 INFO:teuthology.task.internal:Removing archive directory... 2024-04-02T03:13:17.744 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-02T03:13:17.786 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-02T03:13:17.804 INFO:teuthology.task.internal:Not uploading archives. 2024-04-02T03:13:17.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-02T03:13:17.874 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-02T03:13:17.874 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-02T03:13:17.889 INFO:teuthology.orchestra.run.smithi186.stdout: 265426 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 2 03:13 /home/ubuntu/cephtest 2024-04-02T03:13:17.890 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-02T03:13:17.900 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-02T03:13:17.922 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=ci-yingxin-crimson-load-obc-3-base&var-job_id_selected=7635285 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} duration: 997.3525619506836 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-04-02T03:13:17.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-02T03:13:18.003 INFO:teuthology.run:pass