2024-03-29T02:00:46.907 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T02:00:46.907 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041 --name yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} -- /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041/orig.config.yaml 2024-03-29T02:00:46.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T02:00:47.045 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041 branch: ci-yingxin-crimson-load-obc-3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7629041' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-3 env: CRIMSON_COMPAT: '1' sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lSnXJNjIgYu32IsgdrTRsGobfJLQcqzyARwjvF9PUBY5cz+i3r0mfMFAlewc7eXp/4tSwwm4Ba+ugDA7/0oI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_01:54:41 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T02:00:47.045 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa; will attempt to use it 2024-03-29T02:00:47.046 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks 2024-03-29T02:00:47.046 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T02:00:47.048 INFO:teuthology.task.internal:Checking packages... 2024-03-29T02:00:47.071 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '785fc5e8d093374f751c7da3e51e9b56b3718d17' 2024-03-29T02:00:47.071 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T02:00:47.071 INFO:teuthology.packaging:ref: None 2024-03-29T02:00:47.071 INFO:teuthology.packaging:tag: None 2024-03-29T02:00:47.071 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-3 2024-03-29T02:00:47.071 INFO:teuthology.packaging:sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:00:47.072 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-3 2024-03-29T02:00:47.286 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2473.g785fc5e8 2024-03-29T02:00:47.287 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T02:00:47.294 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T02:00:47.294 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T02:00:47.345 INFO:teuthology.task.internal:Saving configuration 2024-03-29T02:00:47.361 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T02:00:47.369 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T02:00:47.394 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 01:56:03.556701', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lSnXJNjIgYu32IsgdrTRsGobfJLQcqzyARwjvF9PUBY5cz+i3r0mfMFAlewc7eXp/4tSwwm4Ba+ugDA7/0oI='} 2024-03-29T02:00:47.394 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T02:00:47.403 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-29T02:00:47.403 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T02:00:47.460 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f79ac671f70>, signals=[15]) 2024-03-29T02:00:47.460 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T02:00:47.466 INFO:teuthology.task.internal:Opening connections... 2024-03-29T02:00:47.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:00:47.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:00:47.575 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T02:00:47.582 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-03-29T02:00:47.645 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-03-29T02:00:47.645 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-03-29T02:00:47.702 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-03-29T02:00:47.702 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-03-29T02:00:47.702 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-03-29T02:00:47.702 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T02:00:47.703 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T02:00:47.704 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-03-29T02:00:47.723 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T02:00:47.732 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T02:00:47.754 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T02:00:47.754 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-03-29T02:00:47.770 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T02:00:47.778 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T02:00:47.778 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-03-29T02:00:47.833 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T02:00:47.863 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T02:00:47.889 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T02:00:47.889 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T02:00:47.890 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T02:00:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:00:47.906 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-29T02:00:47.963 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-03-29T02:00:47.963 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-03-29T02:00:47.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-03-29T02:00:49.958 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T02:00:51.408 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T02:00:52.316 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T02:00:55.308 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-29T02:01:01.984 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-29T02:01:04.759 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 9.8 kB/s | 16 kB 00:01 2024-03-29T02:01:06.127 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T02:01:11.461 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T02:01:13.851 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T02:01:13.851 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T02:01:13.946 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T02:01:13.962 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 76 M 2024-03-29T02:01:13.963 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 123 M 2024-03-29T02:01:13.964 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:01:14.120 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T02:01:14.821 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T02:01:15.929 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-03-29T02:01:16.537 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 11 MB/s | 28 MB 00:02 2024-03-29T02:01:16.540 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:01:16.541 INFO:teuthology.orchestra.run.smithi007.stdout:Total 30 MB/s | 76 MB 00:02 2024-03-29T02:01:16.541 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:01:16.995 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:01:16.995 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:01:17.879 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:01:17.880 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:01:20.308 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:01:20.931 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T02:01:21.024 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:01:23.914 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:01:24.052 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:01:29.125 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:01:29.160 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:01:34.259 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:02:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:02:05.180 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T02:02:05.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:02:05.225 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:02:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:02:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T02:02:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:02:05.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:02:06.024 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:02:06.025 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:02:06.368 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T02:02:06.484 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T02:02:06.484 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T02:02:06.485 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T02:02:06.485 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T02:02:06.485 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T02:02:06.485 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T02:02:06.485 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T02:02:06.485 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T02:02:06.568 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T02:02:06.569 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T02:02:06.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-03-29T02:02:06.611 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-03-29T02:02:06.611 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-03-29T02:02:06.611 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-03-29T02:02:06.611 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-03-29T02:02:06.611 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:02:06.612 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T02:02:06.613 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T02:02:06.613 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T02:02:06.613 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-03-29T02:02:06.613 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:02:06.613 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T02:02:06.614 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T02:02:06.614 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-434.el9.x86_64 2024-03-29T02:02:06.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T02:02:07.220 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-03-29T02:02:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T02:02:08.654 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-03-29T02:02:08.656 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-29T02:02:08.672 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.MyDwQGzCNm 2024-03-29T02:02:08.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MyDwQGzCNm 2024-03-29T02:02:08.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.MyDwQGzCNm 2024-03-29T02:02:08.886 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.MyDwQGzCNm is 6KB 2024-03-29T02:02:08.935 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.MyDwQGzCNm 2024-03-29T02:02:08.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T02:02:09.018 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T02:02:09.019 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T02:02:09.019 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T02:02:09.019 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T02:02:09.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T02:02:09.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-03-29T02:02:39.222 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T02:02:39.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-29T02:02:39.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:03:39.284 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T02:03:48.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-29T02:03:48.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:03:48.582 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-29T02:03:49.033 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-29T02:03:49.034 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T02:03:49.034 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-29T02:03:49.048 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-434.el9.x86_64 2024-03-29T02:03:49.048 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T02:03:49.048 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T02:03:49.048 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T02:03:50.049 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T02:03:50.049 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T02:03:50.100 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-03-29T02:03:50.145 DEBUG:teuthology.parallel:result is None 2024-03-29T02:03:50.145 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T02:03:50.152 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T02:03:50.153 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T02:03:50.167 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T02:03:50.194 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T02:03:50.228 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T02:03:50.228 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T02:03:50.247 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T02:03:50.252 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T02:03:50.253 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T02:03:50.327 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T02:03:50.340 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T02:03:50.341 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T02:03:50.348 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T02:03:50.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T02:03:50.408 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T02:03:50.417 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T02:03:50.417 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T02:03:50.467 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T02:03:50.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T02:03:50.607 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T02:03:51.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T02:03:52.016 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:03:52.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T02:03:52.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-03-29T02:03:52.175 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T02:03:52.596 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T02:03:52.604 INFO:teuthology.task.internal:Starting timer... 2024-03-29T02:03:52.604 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T02:03:52.640 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T02:03:52.673 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-29T02:03:52.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-03-29T02:03:52.745 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-03-29T02:03:52.747 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T02:03:52.747 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-03-29T02:03:52.765 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-03-29T02:03:52.765 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive'} 2024-03-29T02:03:52.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T02:03:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:03:52.837 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T02:03:52.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-03-29T02:03:52.917 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T02:03:52.927 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T02:03:52.942 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T02:03:52.944 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T02:08:11.334 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com')] 2024-03-29T02:08:11.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-29T02:08:11.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:08:11.439 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-29T02:08:11.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-29T02:08:11.518 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T02:08:11.528 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T02:08:11.528 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T02:08:11.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T02:08:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T02:08:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T02:08:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-03-29T02:08:11.649 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-03-29T02:08:11.665 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T02:08:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T02:08:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-29T02:08:11.736 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T02:08:11.736 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-29T02:08:11.736 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:08:11.736 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:08:11.736 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:08:11.736 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:08:11.737 INFO:teuthology.run_tasks:Running task install... 2024-03-29T02:08:11.746 DEBUG:teuthology.task.install:project ceph 2024-03-29T02:08:11.747 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}} 2024-03-29T02:08:11.747 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'} 2024-03-29T02:08:11.747 INFO:teuthology.task.install:Using flavor: crimson 2024-03-29T02:08:11.754 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T02:08:11.754 INFO:teuthology.task.install:extra packages: [] 2024-03-29T02:08:11.754 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'tag': None, 'wait_for_package': False} 2024-03-29T02:08:11.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:08:11.900 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/ 2024-03-29T02:08:11.900 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2473.g785fc5e8 2024-03-29T02:08:12.070 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T02:08:12.070 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:08:12.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T02:08:12.098 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T02:08:12.099 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/785fc5e8d093374f751c7da3e51e9b56b3718d17/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T02:08:12.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T02:08:12.246 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T02:08:12.313 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-03-29T02:08:12.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-03-29T02:08:12.618 INFO:teuthology.orchestra.run.smithi007.stdout:74 files removed 2024-03-29T02:08:12.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-03-29T02:08:13.265 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 278 kB/s | 87 kB 00:00 2024-03-29T02:08:13.616 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 39 kB/s | 11 kB 00:00 2024-03-29T02:08:14.049 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 5.2 kB/s | 2.0 kB 00:00 2024-03-29T02:08:14.677 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T02:08:16.119 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T02:08:16.876 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-29T02:08:18.838 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-29T02:08:23.928 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-29T02:08:25.623 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T02:08:27.074 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T02:08:32.147 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T02:08:34.974 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 9.6 M 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-03-29T02:08:34.978 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.7 M 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.9 M 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 19 M 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 25 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 796 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T02:08:34.979 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 547 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 45 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 134 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 159 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 315 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 293 k 2024-03-29T02:08:34.980 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 99 k 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout:Install 30 Packages 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-03-29T02:08:34.981 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:08:34.982 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 58 M 2024-03-29T02:08:34.982 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:08:36.798 INFO:teuthology.orchestra.run.smithi007.stdout:(1/32): ceph-radosgw-19.0.0-2473.g785fc5e8.el9. 5.5 MB/s | 9.6 MB 00:01 2024-03-29T02:08:36.881 INFO:teuthology.orchestra.run.smithi007.stdout:(2/32): ceph-base-19.0.0-2473.g785fc5e8.el9.x86 3.2 MB/s | 5.9 MB 00:01 2024-03-29T02:08:36.907 INFO:teuthology.orchestra.run.smithi007.stdout:(3/32): ceph-selinux-19.0.0-2473.g785fc5e8.el9. 232 kB/s | 25 kB 00:00 2024-03-29T02:08:37.107 INFO:teuthology.orchestra.run.smithi007.stdout:(4/32): ceph-common-19.0.0-2473.g785fc5e8.el9.x 9.1 MB/s | 19 MB 00:02 2024-03-29T02:08:37.141 INFO:teuthology.orchestra.run.smithi007.stdout:(5/32): libradosstriper1-19.0.0-2473.g785fc5e8. 2.3 MB/s | 547 kB 00:00 2024-03-29T02:08:37.174 INFO:teuthology.orchestra.run.smithi007.stdout:(6/32): libcephfs2-19.0.0-2473.g785fc5e8.el9.x8 2.7 MB/s | 796 kB 00:00 2024-03-29T02:08:37.199 INFO:teuthology.orchestra.run.smithi007.stdout:(7/32): python3-ceph-argparse-19.0.0-2473.g785f 771 kB/s | 45 kB 00:00 2024-03-29T02:08:37.275 INFO:teuthology.orchestra.run.smithi007.stdout:(8/32): python3-cephfs-19.0.0-2473.g785fc5e8.el 2.1 MB/s | 159 kB 00:00 2024-03-29T02:08:37.300 INFO:teuthology.orchestra.run.smithi007.stdout:(9/32): python3-ceph-common-19.0.0-2473.g785fc5 1.0 MB/s | 134 kB 00:00 2024-03-29T02:08:37.358 INFO:teuthology.orchestra.run.smithi007.stdout:(10/32): python3-rados-19.0.0-2473.g785fc5e8.el 3.7 MB/s | 315 kB 00:00 2024-03-29T02:08:37.417 INFO:teuthology.orchestra.run.smithi007.stdout:(11/32): python3-rbd-19.0.0-2473.g785fc5e8.el9. 2.4 MB/s | 293 kB 00:00 2024-03-29T02:08:37.467 INFO:teuthology.orchestra.run.smithi007.stdout:(12/32): python3-rgw-19.0.0-2473.g785fc5e8.el9. 915 kB/s | 99 kB 00:00 2024-03-29T02:08:37.559 INFO:teuthology.orchestra.run.smithi007.stdout:(13/32): librgw2-19.0.0-2473.g785fc5e8.el9.x86_ 11 MB/s | 5.0 MB 00:00 2024-03-29T02:08:37.643 INFO:teuthology.orchestra.run.smithi007.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 469 kB/s | 106 kB 00:00 2024-03-29T02:08:37.660 INFO:teuthology.orchestra.run.smithi007.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 44 kB 00:00 2024-03-29T02:08:37.727 INFO:teuthology.orchestra.run.smithi007.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 683 kB/s | 159 kB 00:00 2024-03-29T02:08:37.785 INFO:teuthology.orchestra.run.smithi007.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-29T02:08:37.810 INFO:teuthology.orchestra.run.smithi007.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-03-29T02:08:37.827 INFO:teuthology.orchestra.run.smithi007.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 412 kB/s | 41 kB 00:00 2024-03-29T02:08:37.852 INFO:teuthology.orchestra.run.smithi007.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T02:08:37.869 INFO:teuthology.orchestra.run.smithi007.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T02:08:37.886 INFO:teuthology.orchestra.run.smithi007.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 597 kB/s | 35 kB 00:00 2024-03-29T02:08:38.070 INFO:teuthology.orchestra.run.smithi007.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-29T02:08:38.204 INFO:teuthology.orchestra.run.smithi007.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-03-29T02:08:38.229 INFO:teuthology.orchestra.run.smithi007.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 821 kB/s | 309 kB 00:00 2024-03-29T02:08:38.337 INFO:teuthology.orchestra.run.smithi007.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-03-29T02:08:38.463 INFO:teuthology.orchestra.run.smithi007.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-03-29T02:08:38.488 INFO:teuthology.orchestra.run.smithi007.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T02:08:38.522 INFO:teuthology.orchestra.run.smithi007.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-03-29T02:08:38.630 INFO:teuthology.orchestra.run.smithi007.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-29T02:08:38.814 INFO:teuthology.orchestra.run.smithi007.stdout:(31/32): librbd1-19.0.0-2473.g785fc5e8.el9.x86_ 13 MB/s | 3.7 MB 00:00 2024-03-29T02:08:39.081 INFO:teuthology.orchestra.run.smithi007.stdout:(32/32): librados2-19.0.0-2473.g785fc5e8.el9.x8 5.7 MB/s | 3.4 MB 00:00 2024-03-29T02:08:39.085 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:08:39.086 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T02:08:39.086 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:08:39.129 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:08:39.129 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:08:39.628 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:08:39.629 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:08:41.176 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:08:41.701 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T02:08:41.741 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:08:41.834 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:08:41.925 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-29T02:08:41.951 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:08:42.013 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:08:42.063 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-29T02:08:42.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-29T02:08:42.114 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-29T02:08:42.136 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-29T02:08:42.152 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-29T02:08:42.358 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-29T02:08:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-29T02:08:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-29T02:08:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:08:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:08:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 12/34 2024-03-29T02:08:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-29T02:08:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 14/34 2024-03-29T02:08:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-29T02:08:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-29T02:08:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-29T02:08:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-29T02:08:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-29T02:08:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-29T02:08:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-29T02:08:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-29T02:08:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-29T02:08:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-29T02:08:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T02:08:44.107 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T02:08:44.133 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-29T02:08:44.197 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-29T02:08:44.275 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 27/34 2024-03-29T02:08:44.309 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 28/34 2024-03-29T02:08:46.871 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:08:46.914 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:08:47.626 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:08:47.676 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-29T02:08:47.773 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-29T02:08:47.773 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T02:08:47.773 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T02:08:47.773 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:08:47.804 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:08:58.416 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:08:58.416 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-03-29T02:08:58.417 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-03-29T02:08:58.417 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-03-29T02:08:58.417 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-03-29T02:08:58.417 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-03-29T02:08:58.418 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-03-29T02:08:58.418 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-03-29T02:08:58.418 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:08:59.453 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-29T02:08:59.478 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-29T02:08:59.478 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:08:59.478 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T02:08:59.478 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T02:08:59.478 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T02:08:59.478 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:08:59.480 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T02:08:59.548 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T02:08:59.548 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 6/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 7/34 2024-03-29T02:09:00.825 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 8/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 9/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 12/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T02:09:00.826 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T02:09:00.827 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/34 2024-03-29T02:09:01.782 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:09:01.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:01.782 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-03-29T02:09:01.782 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.782 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T02:09:01.783 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.784 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:01.785 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:09:02.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-03-29T02:09:02.664 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:09:02.830 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 87 M 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 174 k 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-29T02:09:02.831 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout:Install 11 Packages 2024-03-29T02:09:02.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:02.833 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 90 M 2024-03-29T02:09:02.833 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 819 M 2024-03-29T02:09:02.833 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:09:04.522 INFO:teuthology.orchestra.run.smithi007.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T02:09:04.589 INFO:teuthology.orchestra.run.smithi007.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-29T02:09:04.631 INFO:teuthology.orchestra.run.smithi007.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T02:09:04.665 INFO:teuthology.orchestra.run.smithi007.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T02:09:04.698 INFO:teuthology.orchestra.run.smithi007.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 90 kB 00:00 2024-03-29T02:09:04.823 INFO:teuthology.orchestra.run.smithi007.stdout:(6/11): libcephsqlite-19.0.0-2473.g785fc5e8.el9 346 kB/s | 174 kB 00:00 2024-03-29T02:09:04.857 INFO:teuthology.orchestra.run.smithi007.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T02:09:04.983 INFO:teuthology.orchestra.run.smithi007.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T02:09:05.008 INFO:teuthology.orchestra.run.smithi007.stdout:(9/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.5 MB/s | 469 kB 00:00 2024-03-29T02:09:05.158 INFO:teuthology.orchestra.run.smithi007.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 695 kB/s | 122 kB 00:00 2024-03-29T02:09:09.250 INFO:teuthology.orchestra.run.smithi007.stdout:(11/11): ceph-test-19.0.0-2473.g785fc5e8.el9.x8 18 MB/s | 87 MB 00:04 2024-03-29T02:09:09.252 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:09:09.253 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 90 MB 00:06 2024-03-29T02:09:09.294 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:09:09.320 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:09:09.320 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:09:09.993 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:09:09.993 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:09:11.054 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:09:11.112 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-29T02:09:11.143 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-29T02:09:11.172 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-29T02:09:11.256 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-29T02:09:11.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-29T02:09:11.338 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-29T02:09:11.382 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-29T02:09:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-29T02:09:11.449 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-29T02:09:11.489 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-29T02:09:11.515 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-29T02:09:23.697 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-29T02:09:23.800 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-29T02:09:30.310 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-29T02:09:30.310 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-29T02:09:30.311 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-29T02:09:30.725 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-29T02:09:30.725 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:30.725 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T02:09:30.726 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T02:09:30.727 INFO:teuthology.orchestra.run.smithi007.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-29T02:09:30.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:30.727 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:09:31.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-03-29T02:09:31.615 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:09:31.772 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:09:31.775 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-29T02:09:31.775 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:09:31.775 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-29T02:09:31.775 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:09:31.775 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 6.5 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 2.2 M 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 1.6 M 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 249 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 71 M 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T02:09:31.776 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T02:09:31.777 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-29T02:09:31.778 INFO:teuthology.orchestra.run.smithi007.stdout:Install 36 Packages 2024-03-29T02:09:31.779 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:31.779 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 84 M 2024-03-29T02:09:31.779 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 921 M 2024-03-29T02:09:31.779 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:09:32.573 INFO:teuthology.orchestra.run.smithi007.stdout:(1/36): ceph-19.0.0-2473.g785fc5e8.el9.x86_64.r 14 kB/s | 6.5 kB 00:00 2024-03-29T02:09:33.573 INFO:teuthology.orchestra.run.smithi007.stdout:(2/36): ceph-mgr-19.0.0-2473.g785fc5e8.el9.x86_ 1.1 MB/s | 1.6 MB 00:01 2024-03-29T02:09:33.815 INFO:teuthology.orchestra.run.smithi007.stdout:(3/36): ceph-mds-19.0.0-2473.g785fc5e8.el9.x86_ 1.3 MB/s | 2.2 MB 00:01 2024-03-29T02:09:33.882 INFO:teuthology.orchestra.run.smithi007.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2473.g785f 3.7 MB/s | 249 kB 00:00 2024-03-29T02:09:34.007 INFO:teuthology.orchestra.run.smithi007.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T02:09:34.049 INFO:teuthology.orchestra.run.smithi007.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T02:09:34.083 INFO:teuthology.orchestra.run.smithi007.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-29T02:09:34.117 INFO:teuthology.orchestra.run.smithi007.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T02:09:34.158 INFO:teuthology.orchestra.run.smithi007.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 931 kB/s | 39 kB 00:00 2024-03-29T02:09:34.225 INFO:teuthology.orchestra.run.smithi007.stdout:(10/36): ceph-mon-19.0.0-2473.g785fc5e8.el9.x86 3.0 MB/s | 5.0 MB 00:01 2024-03-29T02:09:34.251 INFO:teuthology.orchestra.run.smithi007.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-29T02:09:34.309 INFO:teuthology.orchestra.run.smithi007.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T02:09:34.335 INFO:teuthology.orchestra.run.smithi007.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T02:09:34.360 INFO:teuthology.orchestra.run.smithi007.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T02:09:34.386 INFO:teuthology.orchestra.run.smithi007.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T02:09:34.477 INFO:teuthology.orchestra.run.smithi007.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-29T02:09:34.694 INFO:teuthology.orchestra.run.smithi007.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-29T02:09:34.761 INFO:teuthology.orchestra.run.smithi007.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 654 kB/s | 43 kB 00:00 2024-03-29T02:09:34.886 INFO:teuthology.orchestra.run.smithi007.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T02:09:34.987 INFO:teuthology.orchestra.run.smithi007.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-29T02:09:35.037 INFO:teuthology.orchestra.run.smithi007.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-29T02:09:35.079 INFO:teuthology.orchestra.run.smithi007.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-03-29T02:09:35.121 INFO:teuthology.orchestra.run.smithi007.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 561 kB/s | 23 kB 00:00 2024-03-29T02:09:35.155 INFO:teuthology.orchestra.run.smithi007.stdout:(24/36): python3-requests-2.25.1-8.el9.noarch.r 157 kB/s | 125 kB 00:00 2024-03-29T02:09:35.171 INFO:teuthology.orchestra.run.smithi007.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-03-29T02:09:35.239 INFO:teuthology.orchestra.run.smithi007.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2024-03-29T02:09:35.264 INFO:teuthology.orchestra.run.smithi007.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-03-29T02:09:35.306 INFO:teuthology.orchestra.run.smithi007.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-29T02:09:35.365 INFO:teuthology.orchestra.run.smithi007.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T02:09:35.432 INFO:teuthology.orchestra.run.smithi007.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T02:09:35.465 INFO:teuthology.orchestra.run.smithi007.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T02:09:35.491 INFO:teuthology.orchestra.run.smithi007.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2024-03-29T02:09:35.541 INFO:teuthology.orchestra.run.smithi007.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T02:09:35.566 INFO:teuthology.orchestra.run.smithi007.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-29T02:09:35.591 INFO:teuthology.orchestra.run.smithi007.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T02:09:37.333 INFO:teuthology.orchestra.run.smithi007.stdout:(36/36): ceph-osd-19.0.0-2473.g785fc5e8.el9.x86 19 MB/s | 71 MB 00:03 2024-03-29T02:09:37.337 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:09:37.337 INFO:teuthology.orchestra.run.smithi007.stdout:Total 15 MB/s | 84 MB 00:05 2024-03-29T02:09:37.362 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:09:37.409 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:09:37.409 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:09:38.187 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:09:38.187 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:09:39.807 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:09:39.869 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T02:09:39.903 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T02:09:39.930 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T02:09:40.006 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T02:09:40.054 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T02:09:40.087 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T02:09:40.185 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T02:09:40.260 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T02:09:40.316 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T02:09:40.391 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T02:09:40.450 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T02:09:40.487 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T02:09:40.554 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T02:09:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T02:09:40.813 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T02:09:40.858 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T02:09:40.897 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T02:09:40.940 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T02:09:40.970 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T02:09:41.051 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T02:09:41.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T02:09:41.122 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T02:09:41.163 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T02:09:41.206 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T02:09:41.249 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T02:09:41.344 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T02:09:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T02:09:41.455 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T02:09:41.632 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T02:09:41.763 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T02:09:41.958 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 31/36 2024-03-29T02:09:41.984 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-29T02:09:42.010 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-29T02:09:42.010 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:09:42.010 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T02:09:42.010 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T02:09:42.011 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T02:09:42.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:53.476 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-29T02:09:53.503 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-29T02:09:53.503 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:09:53.503 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T02:09:53.503 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T02:09:53.503 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T02:09:53.503 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:54.060 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-29T02:09:54.083 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-29T02:09:54.083 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:09:54.083 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T02:09:54.083 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T02:09:54.083 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T02:09:54.083 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:54.327 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-29T02:09:54.350 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-29T02:09:54.351 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:09:54.351 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T02:09:54.351 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T02:09:54.351 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T02:09:54.351 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:09:54.411 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/36 2024-03-29T02:10:01.427 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 6/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T02:10:01.428 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T02:10:01.429 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T02:10:01.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T02:10:02.439 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T02:10:02.440 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T02:10:02.441 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:02.442 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:02.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-03-29T02:10:03.277 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:03.379 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:03.436 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:03.437 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:03.438 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:03.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-03-29T02:10:04.094 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:04.251 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 762 k 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:04.252 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:04.253 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:04.253 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:10:04.253 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:04.253 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 762 k 2024-03-29T02:10:04.253 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 768 k 2024-03-29T02:10:04.253 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:05.040 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-19.0.0-2473.g785fc5e8.el9.noarch.rpm 968 kB/s | 762 kB 00:00 2024-03-29T02:10:05.041 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:05.041 INFO:teuthology.orchestra.run.smithi007.stdout:Total 966 kB/s | 762 kB 00:00 2024-03-29T02:10:05.041 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:05.047 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:05.047 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:05.133 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:05.526 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:10:05.614 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:10:06.020 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:10:06.308 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:10:06.308 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:06.308 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:06.308 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:06.308 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:06.308 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:06.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T02:10:07.098 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:07.254 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 157 k 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:07.255 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:07.256 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:07.256 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:10:07.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:07.256 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 157 k 2024-03-29T02:10:07.256 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 640 k 2024-03-29T02:10:07.256 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:07.739 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-19.0.0-2473.g785fc5 326 kB/s | 157 kB 00:00 2024-03-29T02:10:07.740 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:07.740 INFO:teuthology.orchestra.run.smithi007.stdout:Total 324 kB/s | 157 kB 00:00 2024-03-29T02:10:07.740 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:07.750 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:07.750 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:07.806 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:07.807 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:07.951 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:08.006 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:10:08.031 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:10:08.031 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:10:08.031 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T02:10:08.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:09.033 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:10:09.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:09.033 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:09.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:09.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-03-29T02:10:09.875 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:09.979 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:10.037 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:10.038 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:10.038 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:10.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T02:10:10.686 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:10.842 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:10.843 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 3.5 M 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 34 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 16 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T02:10:10.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout:Install 11 Packages 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 7.6 M 2024-03-29T02:10:10.845 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 100 M 2024-03-29T02:10:10.846 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:11.243 INFO:teuthology.orchestra.run.smithi007.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2473.g78 175 kB/s | 34 kB 00:00 2024-03-29T02:10:11.335 INFO:teuthology.orchestra.run.smithi007.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2473.g785 58 kB/s | 16 kB 00:00 2024-03-29T02:10:11.368 INFO:teuthology.orchestra.run.smithi007.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-29T02:10:11.511 INFO:teuthology.orchestra.run.smithi007.stdout:(4/11): ceph-mgr-dashboard-19.0.0-2473.g785fc5e 7.7 MB/s | 3.5 MB 00:00 2024-03-29T02:10:11.635 INFO:teuthology.orchestra.run.smithi007.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-03-29T02:10:11.694 INFO:teuthology.orchestra.run.smithi007.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T02:10:11.761 INFO:teuthology.orchestra.run.smithi007.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-29T02:10:11.820 INFO:teuthology.orchestra.run.smithi007.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 526 kB/s | 31 kB 00:00 2024-03-29T02:10:11.887 INFO:teuthology.orchestra.run.smithi007.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-03-29T02:10:12.021 INFO:teuthology.orchestra.run.smithi007.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.3 MB/s | 2.0 MB 00:00 2024-03-29T02:10:13.037 INFO:teuthology.orchestra.run.smithi007.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 507 kB/s | 863 kB 00:01 2024-03-29T02:10:13.042 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:13.042 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.5 MB/s | 7.6 MB 00:02 2024-03-29T02:10:13.063 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:13.082 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:13.083 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:13.269 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:13.270 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:13.791 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-29T02:10:13.899 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-29T02:10:14.007 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-29T02:10:14.147 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-29T02:10:14.194 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-29T02:10:14.283 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-29T02:10:14.331 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-29T02:10:14.366 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-29T02:10:14.407 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 9/11 2024-03-29T02:10:15.514 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 10/11 2024-03-29T02:10:15.557 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-29T02:10:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-29T02:10:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 1/11 2024-03-29T02:10:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 2/11 2024-03-29T02:10:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 3/11 2024-03-29T02:10:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-29T02:10:16.860 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-29T02:10:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-29T02:10:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-29T02:10:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-29T02:10:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-29T02:10:16.861 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-29T02:10:17.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T02:10:17.208 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T02:10:17.209 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T02:10:17.209 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:17.209 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:17.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T02:10:18.051 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:18.209 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 7.4 M 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:10:18.210 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:18.211 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:18.212 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-29T02:10:18.212 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-03-29T02:10:18.212 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:18.212 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2024-03-29T02:10:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2024-03-29T02:10:18.213 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:18.920 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 202 kB/s | 32 kB 00:00 2024-03-29T02:10:18.945 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 698 kB/s | 17 kB 00:00 2024-03-29T02:10:18.971 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-29T02:10:19.038 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T02:10:19.222 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T02:10:19.288 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T02:10:19.313 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T02:10:19.581 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-24 9.1 MB/s | 7.4 MB 00:00 2024-03-29T02:10:19.747 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-29T02:10:19.858 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-29T02:10:19.948 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 536 kB/s | 195 kB 00:00 2024-03-29T02:10:20.148 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 613 kB/s | 245 kB 00:00 2024-03-29T02:10:20.152 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:20.152 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-29T02:10:20.192 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:20.217 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:20.217 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:20.592 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:20.593 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:21.322 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:21.405 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T02:10:21.444 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T02:10:21.543 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T02:10:21.589 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T02:10:21.777 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T02:10:21.805 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T02:10:21.894 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T02:10:22.588 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T02:10:22.707 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T02:10:24.055 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T02:10:25.129 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T02:10:25.155 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-29T02:10:27.676 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 1/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T02:10:27.677 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T02:10:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T02:10:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:28.084 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T02:10:28.084 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:28.085 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:28.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-03-29T02:10:29.072 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:29.229 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:29.230 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-29T02:10:29.230 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:29.230 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-29T02:10:29.230 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 49 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T02:10:29.231 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-03-29T02:10:29.232 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:29.541 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 129 kB/s | 18 kB 00:00 2024-03-29T02:10:29.565 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-03-29T02:10:29.616 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T02:10:29.641 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T02:10:29.667 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T02:10:29.692 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T02:10:29.725 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): ceph-mgr-rook-19.0.0-2473.g785fc5e8.el9 152 kB/s | 49 kB 00:00 2024-03-29T02:10:29.918 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-03-29T02:10:29.934 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-29T02:10:30.093 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-03-29T02:10:30.160 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-29T02:10:30.252 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 565 kB/s | 90 kB 00:00 2024-03-29T02:10:30.444 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T02:10:30.446 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:30.446 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-29T02:10:30.446 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:30.462 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:30.462 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:30.561 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:30.561 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:31.088 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:31.291 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T02:10:31.350 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T02:10:31.405 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T02:10:31.440 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T02:10:31.483 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T02:10:31.582 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T02:10:31.700 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T02:10:31.751 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T02:10:32.520 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T02:10:32.572 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T02:10:32.606 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T02:10:32.655 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T02:10:32.684 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 1/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T02:10:33.487 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T02:10:33.488 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T02:10:33.488 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T02:10:33.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:33.959 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:34.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T02:10:34.797 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:34.957 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:34.957 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-29T02:10:34.957 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 141 k 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T02:10:34.958 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T02:10:34.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:34.959 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:34.959 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-29T02:10:34.959 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-03-29T02:10:34.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:34.960 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2024-03-29T02:10:34.960 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2024-03-29T02:10:34.960 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:35.340 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-03-29T02:10:35.498 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-29T02:10:35.715 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2473.g785fc5e8.e 266 kB/s | 141 kB 00:00 2024-03-29T02:10:35.741 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 238 kB/s | 58 kB 00:00 2024-03-29T02:10:35.774 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T02:10:35.941 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 226 kB/s | 51 kB 00:00 2024-03-29T02:10:35.943 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:35.943 INFO:teuthology.orchestra.run.smithi007.stdout:Total 6.9 MB/s | 6.8 MB 00:00 2024-03-29T02:10:35.956 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:35.967 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:35.967 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:36.060 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:36.060 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:36.366 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:36.539 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T02:10:36.593 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T02:10:37.148 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T02:10:37.231 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T02:10:37.311 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T02:10:37.338 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-29T02:10:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-29T02:10:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/6 2024-03-29T02:10:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T02:10:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T02:10:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T02:10:38.194 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T02:10:38.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T02:10:38.504 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:38.505 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:38.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-03-29T02:10:39.357 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:39.515 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:39.515 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 871 k 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-29T02:10:39.516 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:39.517 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 949 k 2024-03-29T02:10:39.517 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 4.0 M 2024-03-29T02:10:39.517 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:39.643 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 624 kB/s | 79 kB 00:00 2024-03-29T02:10:39.852 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-19.0.0-2473.g785fc5e8.el9.x86_ 2.5 MB/s | 871 kB 00:00 2024-03-29T02:10:39.853 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:39.853 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.8 MB/s | 949 kB 00:00 2024-03-29T02:10:39.853 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:39.878 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:39.878 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:39.956 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:39.956 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:40.152 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:40.291 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T02:10:40.363 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-29T02:10:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-29T02:10:41.090 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/2 2024-03-29T02:10:41.395 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T02:10:41.395 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:41.395 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:41.395 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T02:10:41.395 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:41.395 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:41.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-03-29T02:10:42.245 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:10 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:42.406 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 264 k 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-29T02:10:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:42.408 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 332 k 2024-03-29T02:10:42.408 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.5 M 2024-03-29T02:10:42.408 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:42.533 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 557 kB/s | 69 kB 00:00 2024-03-29T02:10:42.807 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-volume-19.0.0-2473.g785fc5e8.el9.no 662 kB/s | 264 kB 00:00 2024-03-29T02:10:42.808 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:42.809 INFO:teuthology.orchestra.run.smithi007.stdout:Total 829 kB/s | 332 kB 00:00 2024-03-29T02:10:42.809 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:42.818 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:42.818 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:42.861 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:42.861 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T02:10:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-29T02:10:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-29T02:10:43.334 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:10:43.335 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T02:10:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:44.079 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 1/2 2024-03-29T02:10:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T02:10:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:44.412 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:10:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T02:10:44.413 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:44.413 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:44.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-03-29T02:10:45.306 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:13 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:45.464 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:45.465 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:45.465 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-29T02:10:45.465 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:45.465 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 125 k 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 125 k 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 445 k 2024-03-29T02:10:45.466 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:45.689 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-19.0.0-2473.g785fc5e8.el9.x86_64 560 kB/s | 125 kB 00:00 2024-03-29T02:10:45.690 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:45.690 INFO:teuthology.orchestra.run.smithi007.stdout:Total 556 kB/s | 125 kB 00:00 2024-03-29T02:10:45.690 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:45.696 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:45.696 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:45.754 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:45.754 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:45.904 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:46.025 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:46.462 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:46.744 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:46.744 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:46.744 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:46.744 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:46.744 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:46.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-03-29T02:10:47.532 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:47.634 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:47.689 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:47.691 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:47.691 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:47.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-03-29T02:10:48.340 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:16 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:48.498 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 32 k 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:48.499 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:10:48.500 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:48.500 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 32 k 2024-03-29T02:10:48.500 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 142 k 2024-03-29T02:10:48.500 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-19.0.0-2473.g785fc5e8.el9.x86_6 82 kB/s | 32 kB 00:00 2024-03-29T02:10:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:Total 82 kB/s | 32 kB 00:00 2024-03-29T02:10:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:48.890 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:48.929 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:48.929 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:49.025 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:49.144 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:49.532 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:49.793 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:49.793 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:49.793 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:49.794 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:49.794 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:49.794 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:49.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-03-29T02:10:50.540 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:50.642 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:50.696 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:50.698 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:50.698 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:50.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-03-29T02:10:51.343 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:19 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:51.443 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:51.498 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:51.499 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:51.499 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:51.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-03-29T02:10:52.146 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:52.248 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:52.303 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:52.304 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:52.304 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:52.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-03-29T02:10:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:53.053 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:53.108 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:53.109 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:53.109 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:53.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-03-29T02:10:53.751 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:53.852 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:53.907 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:53.908 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:53.908 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:53.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-03-29T02:10:54.549 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:54.652 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:10:54.709 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:54.710 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-29T02:10:54.710 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:54.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-03-29T02:10:55.362 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:55.518 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:55.519 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:55.519 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-29T02:10:55.519 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:55.519 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:55.519 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 90 k 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 90 k 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 313 k 2024-03-29T02:10:55.520 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:55.872 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 255 kB/s | 90 kB 00:00 2024-03-29T02:10:55.872 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:55.872 INFO:teuthology.orchestra.run.smithi007.stdout:Total 254 kB/s | 90 kB 00:00 2024-03-29T02:10:55.872 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:55.882 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:55.882 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:55.940 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:55.940 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:10:56.098 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:10:56.215 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:56.586 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:56.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:10:56.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:56.894 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:10:56.894 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:10:56.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:56.894 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:10:57.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-03-29T02:10:57.720 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:10:57.878 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:10:57.879 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:57.879 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-29T02:10:57.879 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.4 M 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 29 M 2024-03-29T02:10:57.880 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:10:59.526 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 2.1 MB/s | 3.4 MB 00:01 2024-03-29T02:10:59.526 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:10:59.527 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.1 MB/s | 3.4 MB 00:01 2024-03-29T02:10:59.527 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:10:59.541 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:10:59.541 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:10:59.623 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:11:00.220 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:11:00.293 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:11:00.318 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:11:00.318 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:11:00.319 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T02:11:00.319 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T02:11:00.319 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T02:11:00.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:01.573 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:11:01.573 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:01.573 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:11:01.573 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:11:01.573 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:01.574 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:11:01.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-03-29T02:11:02.423 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:11:02.580 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:11:02.581 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:11:02.581 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-29T02:11:02.581 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:11:02.581 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:11:02.581 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 176 k 2024-03-29T02:11:02.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 176 k 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 673 k 2024-03-29T02:11:02.582 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:11:02.879 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 592 kB/s | 176 kB 00:00 2024-03-29T02:11:02.880 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:11:02.880 INFO:teuthology.orchestra.run.smithi007.stdout:Total 589 kB/s | 176 kB 00:00 2024-03-29T02:11:02.880 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:11:02.891 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:11:02.891 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:11:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:11:02.950 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:11:03.098 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:11:03.206 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:11:03.666 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:11:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:11:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:03.958 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:11:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:11:03.958 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:03.958 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:11:04.225 DEBUG:teuthology.parallel:result is None 2024-03-29T02:11:04.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:11:04.454 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T02:11:04.475 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-2473.g785fc5e8.el9 2024-03-29T02:11:04.475 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2473.g785fc5e8.el9 2024-03-29T02:11:04.475 INFO:teuthology.task.install:The correct ceph version 19.0.0-2473.g785fc5e8 is installed. 2024-03-29T02:11:04.477 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T02:11:04.495 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:04.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T02:11:04.542 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T02:11:04.543 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:04.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T02:11:04.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T02:11:04.676 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T02:11:04.677 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:04.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T02:11:04.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T02:11:04.810 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T02:11:04.811 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:04.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T02:11:04.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T02:11:04.945 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T02:11:05.237 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T02:11:05.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-03-29T02:11:05.264 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T02:11:05.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T02:11:05.331 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T02:11:05.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T02:11:05.408 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T02:11:05.408 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-29T02:11:05.409 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041', 'branch': 'ci-yingxin-crimson-load-obc-3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7629041', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+lSnXJNjIgYu32IsgdrTRsGobfJLQcqzyARwjvF9PUBY5cz+i3r0mfMFAlewc7eXp/4tSwwm4Ba+ugDA7/0oI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-29_01:54:41', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T02:11:05.409 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T02:11:05.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T02:11:05.521 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:05.521 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T02:11:05.576 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T02:11:05.576 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-03-29T02:11:05.630 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T02:11:05.630 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:11:05.630 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T02:11:05.630 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:11:05.631 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:11:05.631 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-29 02:11:00.570581890 +0000 2024-03-29T02:11:05.631 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-29 02:07:47.817620393 +0000 2024-03-29T02:11:05.631 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-29 02:07:47.817620393 +0000 2024-03-29T02:11:05.631 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-29 02:07:47.817620393 +0000 2024-03-29T02:11:05.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T02:11:05.696 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-29T02:11:05.696 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-29T02:11:05.697 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.6633e-05 s, 5.3 MB/s 2024-03-29T02:11:05.697 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T02:11:05.753 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-03-29T02:11:05.808 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T02:11:05.808 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:11:05.808 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T02:11:05.808 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:11:05.809 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:11:05.809 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-29 02:11:00.570581890 +0000 2024-03-29T02:11:05.809 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-29 02:07:48.158615207 +0000 2024-03-29T02:11:05.809 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-29 02:07:48.158615207 +0000 2024-03-29T02:11:05.809 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-29 02:07:48.158615207 +0000 2024-03-29T02:11:05.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T02:11:05.874 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-29T02:11:05.874 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-29T02:11:05.874 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000106799 s, 4.8 MB/s 2024-03-29T02:11:05.875 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T02:11:05.930 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-29 02:11:00.571581875 +0000 2024-03-29T02:11:05.985 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-29 02:07:48.487610204 +0000 2024-03-29T02:11:05.986 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-29 02:07:48.487610204 +0000 2024-03-29T02:11:05.986 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-29 02:07:48.487610204 +0000 2024-03-29T02:11:05.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T02:11:06.049 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-29T02:11:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-29T02:11:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.2977e-05 s, 5.5 MB/s 2024-03-29T02:11:06.051 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T02:11:06.106 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-29 02:11:00.571581875 +0000 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-29 02:07:48.823605094 +0000 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-29 02:07:48.823605094 +0000 2024-03-29T02:11:06.161 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-29 02:07:48.823605094 +0000 2024-03-29T02:11:06.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T02:11:06.226 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-29T02:11:06.226 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-29T02:11:06.226 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.1348e-05 s, 5.6 MB/s 2024-03-29T02:11:06.227 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T02:11:06.283 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:11:06.283 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-29T02:11:06.283 INFO:tasks.ceph:Generating config... 2024-03-29T02:11:06.294 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T02:11:06.294 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T02:11:06.294 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T02:11:06.294 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] log to file = False 2024-03-29T02:11:06.295 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-29T02:11:06.296 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T02:11:06.296 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T02:11:06.296 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T02:11:06.296 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T02:11:06.296 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T02:11:06.296 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] auth client required = none 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] auth cluster required = none 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] auth service required = none 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] auth supported = none 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] debug asok = 0/0 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] debug auth = 0/0 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-03-29T02:11:06.296 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug client = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug context = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug crush = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug filer = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-03-29T02:11:06.297 INFO:tasks.ceph:[global] debug journal = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mds = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug mon = 0/0 2024-03-29T02:11:06.298 INFO:tasks.ceph:[global] debug monc = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug ms = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug osd = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug rados = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-03-29T02:11:06.299 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug refs = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug striper = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug timer = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] debug tp = 0/0 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-29T02:11:06.300 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-29T02:11:06.301 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-29T02:11:06.301 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-29T02:11:06.301 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T02:11:06.301 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T02:11:06.301 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-03-29T02:11:06.301 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-03-29T02:11:06.301 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-03-29T02:11:06.301 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-29T02:11:06.301 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-29T02:11:06.301 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-29T02:11:06.301 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-29T02:11:06.301 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-29T02:11:06.301 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T02:11:06.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T02:11:06.391 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T02:11:06.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T02:11:06.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T02:11:06.472 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7')] 2024-03-29T02:11:06.473 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-29T02:11:06.474 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:06.474 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T02:11:06.527 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:06.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:06.618+0000 7f3f15262f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:06.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:06.618+0000 7f3f15262f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:06.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:06.618+0000 7f3f15262f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:06.619 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:06.619 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid c741a814-a48d-480f-977e-db0279ef4ebb 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:fsid c741a814-a48d-480f-977e-db0279ef4ebb 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-03-29T02:11:06.619969+0000 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-03-29T02:11:06.619969+0000 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 17 (quincy) 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-29T02:11:06.620 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-29T02:11:06.621 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T02:11:06.636 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c741a814-a48d-480f-977e-db0279ef4ebb... 2024-03-29T02:11:06.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = none 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-03-29T02:11:06.726 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.727 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-03-29T02:11:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-03-29T02:11:06.729 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = 172.21.15.7 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: auth client required = none 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: auth cluster required = none 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: auth service required = none 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: debug asok = 0/0 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 0/0 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: debug bdev = 0/0 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 0/0 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 0/0 2024-03-29T02:11:06.730 INFO:teuthology.orchestra.run.smithi007.stdout: debug buffer = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug civetweb = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug client = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug compressor = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug context = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug crush = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug crypto = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug dpdk = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug eventtrace = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug filer = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug filestore = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug finisher = 0/0 2024-03-29T02:11:06.731 INFO:teuthology.orchestra.run.smithi007.stdout: debug fuse = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug heartbeatmap = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug javaclient = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug journal = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug journaler = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug kstore = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug leveldb = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug lockdep = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds balancer = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds locker = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds log = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds log expire = 0/0 2024-03-29T02:11:06.732 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds migrator = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug memdb = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgrc = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug objclass = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug objectcacher = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug objecter = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug optracker = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 0/0 2024-03-29T02:11:06.733 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug perfcounter = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rados = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd mirror = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd replay = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug refs = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw lifecycle = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw notification = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw sync = 0/0 2024-03-29T02:11:06.734 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 0/0 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: debug striper = 0/0 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: debug throttle = 0/0 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: debug timer = 0/0 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: debug tp = 0/0 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: ms client mode = crc 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: ms cluster mode = crc 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: ms mon client mode = crc 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: ms mon cluster mode = crc 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: ms mon service mode = crc 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: ms service mode = crc 2024-03-29T02:11:06.735 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = c741a814-a48d-480f-977e-db0279ef4ebb 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-03-29T02:11:06.736 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-03-29T02:11:06.737 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression algorithm = snappy 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression mode = aggressive 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore rocksdb cf = False 2024-03-29T02:11:06.738 INFO:teuthology.orchestra.run.smithi007.stdout: crimson alien thread cpu cores = 6-7 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: crimson osd obc lru size = 10 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 20 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 20 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug filestore = 0/0 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug journal = 0/0 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 0/0 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 0/0 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: err to stderr = True 2024-03-29T02:11:06.739 INFO:teuthology.orchestra.run.smithi007.stdout: log flush on exit = True 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: log to file = False 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: log to stderr = True 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: osd shutdown pgref assert = True 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-03-29T02:11:06.740 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 0/0 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 0/0 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 0/0 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-03-29T02:11:06.741 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T02:11:06.742 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default crimson = True 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T02:11:06.743 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout: crimson seastar cpu cores = 0-2 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout: crimson seastar cpu cores = 3-5 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout: crimson seastar cpu cores = 0-2 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2024-03-29T02:11:06.744 INFO:teuthology.orchestra.run.smithi007.stdout: crimson seastar cpu cores = 3-5 2024-03-29T02:11:06.745 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T02:11:06.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T02:11:06.810 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T02:11:06.811 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:06.811 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T02:11:06.825 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:06.825 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T02:11:06.880 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:11:06.880 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:06.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T02:11:06.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T02:11:06.960 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:06.961 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:07.015 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T02:11:07.016 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T02:11:07.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T02:11:07.119 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T02:11:07.122 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T02:11:07.122 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T02:11:07.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T02:11:07.172 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T02:11:07.188 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T02:11:07.188 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-29T02:11:07.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T02:11:07.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:11:07.215 INFO:tasks.ceph:role: osd.0 2024-03-29T02:11:07.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:11:07.215 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T02:11:07.282 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:11:07.368 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:11:07.370 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-29T02:11:07.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-29T02:11:07.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T02:11:07.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T02:11:07.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T02:11:07.599 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:11:07.599 INFO:tasks.ceph:role: osd.1 2024-03-29T02:11:07.599 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:11:07.599 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T02:11:07.666 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:11:07.666 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:11:07.666 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:11:07.666 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:11:07.667 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:11:07.667 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:11:07.667 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:11:07.667 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:11:07.667 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:11:07.667 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:11:07.699 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-29T02:11:07.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-29T02:11:07.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T02:11:07.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T02:11:07.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T02:11:07.911 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:11:07.911 INFO:tasks.ceph:role: osd.2 2024-03-29T02:11:07.911 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:11:07.911 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T02:11:07.977 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:11:07.977 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:11:07.977 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:11:07.978 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:11:08.009 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-29T02:11:08.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-29T02:11:08.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T02:11:08.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T02:11:08.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:08.501 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:11:08.502 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2024-03-29 02:11:08,503 seastar - Reactor backend: linux-aio 2024-03-29T02:11:08.526 INFO:teuthology.orchestra.run.smithi007.stderr:==29034==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:11:08.669 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG 2024-03-29 02:11:08,671 [shard 0:main] seastar - reactor::drain 2024-03-29T02:11:09.243 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:09.243 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2024-03-29T02:11:09.243 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2024-03-29T02:11:09.243 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2024-03-29T02:11:09.243 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:09.243 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:09.295 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:11:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2024-03-29 02:11:09,301 seastar - Reactor backend: linux-aio 2024-03-29T02:11:09.308 INFO:teuthology.orchestra.run.smithi007.stderr:==29032==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:11:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x5d51ad7 0x5d567d4 0x5d59bd4 0x5d5a0d6 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x362658a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:11:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2024-03-29T02:11:13.991 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:13.991 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2024-03-29T02:11:13.991 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2024-03-29T02:11:13.992 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2024-03-29T02:11:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:13.992 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:13.992 INFO:teuthology.orchestra.run.smithi007.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid c741a814-a48d-480f-977e-db0279ef4ebb 2024-03-29T02:11:14.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T02:11:14.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:14.337 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:11:14.338 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2024-03-29 02:11:14,340 seastar - Reactor backend: linux-aio 2024-03-29T02:11:14.343 INFO:teuthology.orchestra.run.smithi007.stderr:==29217==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:11:14.451 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0xec5fb 0xe0ec6 0x346a0c2 0x346a4be 0x346a5d7 0x5d5a784 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x3785266 0x3785dbf 0x3625d7a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:11:14.451 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2024-03-29T02:11:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG 2024-03-29 02:11:14,489 [shard 0:main] seastar - reactor::drain 2024-03-29T02:11:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2024-03-29T02:11:15.033 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2024-03-29T02:11:15.033 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2024-03-29T02:11:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:15.033 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:11:15.089 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2024-03-29 02:11:15,091 seastar - Reactor backend: linux-aio 2024-03-29T02:11:15.099 INFO:teuthology.orchestra.run.smithi007.stderr:==29215==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:11:15.164 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x105a85 0x2d93f 0x29edb 0xdfdb3 0x343edf3 0x343effc 0x346c66d 0x365a0e4 0x3f6d8dd 0x5d5994e 0x5d59ea4 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x362658a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:11:15.165 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2024-03-29T02:11:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2024-03-29T02:11:19.786 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2024-03-29T02:11:19.786 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2024-03-29T02:11:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:19.787 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:19.787 INFO:teuthology.orchestra.run.smithi007.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid c741a814-a48d-480f-977e-db0279ef4ebb 2024-03-29T02:11:19.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T02:11:19.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:20.128 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:11:20.129 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2024-03-29 02:11:20,130 seastar - Reactor backend: linux-aio 2024-03-29T02:11:20.134 INFO:teuthology.orchestra.run.smithi007.stderr:==29399==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:11:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x36931 0x363c5de 0x5d7ec68 0x5d7ee71 0x5ddadf4 0x5d59970 0x5d59b4a 0x5d5a0d6 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x3785266 0x3785dbf 0x3625d7a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:11:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2024-03-29T02:11:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:DEBUG 2024-03-29 02:11:20,278 [shard 0:main] seastar - reactor::drain 2024-03-29T02:11:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2024-03-29T02:11:20.825 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2024-03-29T02:11:20.825 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2024-03-29T02:11:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:20.825 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:20.891 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:11:20.894 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2024-03-29 02:11:20,896 seastar - Reactor backend: linux-aio 2024-03-29T02:11:20.909 INFO:teuthology.orchestra.run.smithi007.stderr:==29397==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:11:20.974 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x172f7 0x174bb 0x2eab 0x4a09d 0x35b7c 0x3629e 0x34377b9 0x5d7e67f 0x5d7ea84 0x5d7ee66 0x5ddadf4 0x5d59970 0x5d59ea4 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x362658a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:11:20.974 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:25.569 INFO:teuthology.orchestra.run.smithi007.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid c741a814-a48d-480f-977e-db0279ef4ebb 2024-03-29T02:11:25.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T02:11:25.619 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T02:11:25.619 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:25.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T02:11:25.686 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:25.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T02:11:25.752 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:25.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T02:11:25.818 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:25.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T02:11:25.886 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:25.886 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T02:11:25.941 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T02:11:25.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T02:11:26.007 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-03-29T02:11:26.007 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQA7IwZmvXk0BxAA1oQ41jDzYxygenjYFq6lNA== 2024-03-29T02:11:26.007 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-03-29T02:11:26.007 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQA9IwZme1UIHBAA7HSBr9CJyqswcZ7+s+3QQQ== 2024-03-29T02:11:26.007 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-03-29T02:11:26.008 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBDIwZmEbyvDxAA1YMI622u714x1NKtP3D6dA== 2024-03-29T02:11:26.008 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-03-29T02:11:26.008 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBJIwZm0nsbBBAAy+F6P5IPfwqWa/liAwuLLg== 2024-03-29T02:11:26.008 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-29T02:11:26.008 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQA7IwZm551kChAAizNfPa5BU4Gm1631At0qBQ== 2024-03-29T02:11:26.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T02:11:26.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:11:26.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:11:26.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:11:26.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T02:11:26.363 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T02:11:26.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T02:11:26.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T02:11:26.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:26.501+0000 7f5d14c5ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:26.501+0000 7f5d14c5ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T02:11:26.542 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:11:26.596 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T02:11:26.597 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T02:11:26.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T02:11:26.640 INFO:tasks.ceph.mon.a:Started 2024-03-29T02:11:26.640 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T02:11:26.641 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T02:11:26.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T02:11:26.643 INFO:tasks.ceph.mgr.x:Started 2024-03-29T02:11:26.644 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T02:11:26.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T02:11:26.726 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-29T02:11:26.726+0000 7f48bcea3a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.726 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-29T02:11:26.726+0000 7f48bcea3a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.726 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-29T02:11:26.726+0000 7f48bcea3a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.729 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:26.729+0000 7f1fd3834000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:26.756+0000 7f56466ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:26.763+0000 7f56466ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.833 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-03-29T02:11:26.845 INFO:tasks.ceph:check_enable_crimson: True 2024-03-29T02:11:26.845 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-29T02:11:26.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-29T02:11:26.932 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:26.932+0000 7f1fd3834000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.932 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:26.933+0000 7f1fd3834000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:26.952+0000 7f25b00ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:26.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:26.959+0000 7f25b00ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:27.039 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-29T02:11:27.040+0000 7f48b3bf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-29T02:11:27.040 INFO:tasks.ceph.mon.a.smithi007.stderr:Please be aware that this feature is experimental, untested, 2024-03-29T02:11:27.040 INFO:tasks.ceph.mon.a.smithi007.stderr:unsupported, and may result in data corruption, data loss, 2024-03-29T02:11:27.040 INFO:tasks.ceph.mon.a.smithi007.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-29T02:11:27.040 INFO:tasks.ceph.mon.a.smithi007.stderr:feature with important data. 2024-03-29T02:11:27.040 INFO:tasks.ceph.mon.a.smithi007.stderr: 2024-03-29T02:11:27.422 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:27.423+0000 7f1fd3834000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T02:11:27.478 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:27.479+0000 7f1fd3834000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T02:11:27.846 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T02:11:27.846 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:27.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T02:11:27.873 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:27.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T02:11:27.941 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:27.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T02:11:28.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new 47c13e0f-4598-4c0e-ae31-a0fd154a39cb 0 2024-03-29T02:11:28.035 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.036+0000 7f1fd3834000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T02:11:28.093 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.094+0000 7f1fd3834000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T02:11:28.142 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.143+0000 7f1fd3834000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T02:11:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:28.143+0000 7f003cd14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:28.150+0000 7f003cd14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:28.210 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.210+0000 7f1fd3834000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T02:11:28.229 INFO:teuthology.orchestra.run.smithi007.stdout:0 2024-03-29T02:11:28.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new a699595d-34ab-499b-9e0a-85cf9c65fc1f 1 2024-03-29T02:11:28.254 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.256+0000 7f1fd3834000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T02:11:28.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:28.358+0000 7fab75b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:28.365+0000 7fab75b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:28.381 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.382+0000 7f1fd3834000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T02:11:28.444 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-29T02:11:28.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new ffc584f3-784e-47a8-919b-4223ac892919 2 2024-03-29T02:11:28.455 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.456+0000 7f1fd3834000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T02:11:28.505 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.506+0000 7f1fd3834000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T02:11:28.562 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.563+0000 7f1fd3834000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T02:11:28.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:28.563+0000 7f2270a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:28.571+0000 7f2270a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:28.612 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:28.613+0000 7f1fd3834000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T02:11:28.648 INFO:teuthology.orchestra.run.smithi007.stdout:2 2024-03-29T02:11:28.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-29T02:11:28.691 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T02:11:28.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-29T02:11:28.733 INFO:tasks.ceph.osd.0:Started 2024-03-29T02:11:28.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-29T02:11:28.767 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T02:11:28.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-29T02:11:28.809 INFO:tasks.ceph.osd.1:Started 2024-03-29T02:11:28.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-29T02:11:28.843 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T02:11:28.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-29T02:11:28.887 INFO:tasks.ceph.osd.2:Started 2024-03-29T02:11:28.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:11:29.053 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:29.054+0000 7f1fd3834000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T02:11:29.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:29.052+0000 7f14d9ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:29.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:29.060+0000 7f14d9ced640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:29.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:29.132 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":6,"fsid":"c741a814-a48d-480f-977e-db0279ef4ebb","created":"2024-03-29T02:11:26.745021+0000","modified":"2024-03-29T02:11:28.649749+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T02:11:28.649749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"47c13e0f-4598-4c0e-ae31-a0fd154a39cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a699595d-34ab-499b-9e0a-85cf9c65fc1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ffc584f3-784e-47a8-919b-4223ac892919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:11:29.143 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T02:11:29.144 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T02:11:29.267 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:29.268+0000 7f1fd3834000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T02:11:29.446 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:11:29.459 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:29.461+0000 7f1fd3834000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T02:11:29.544 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:29.545+0000 7f1fd3834000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T02:11:29.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-29T02:11:29.546+0000 7fdfba22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:29.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-29T02:11:29.552+0000 7fdfba22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:29.618 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-29T02:11:29.618 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":6,"fsid":"c741a814-a48d-480f-977e-db0279ef4ebb","created":"2024-03-29T02:11:26.745021+0000","modified":"2024-03-29T02:11:28.649749+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T02:11:28.649749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"47c13e0f-4598-4c0e-ae31-a0fd154a39cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a699595d-34ab-499b-9e0a-85cf9c65fc1f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ffc584f3-784e-47a8-919b-4223ac892919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:11:29.628 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-29T02:11:29.759 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:29.760+0000 7f1fd3834000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T02:11:29.927 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:29.928+0000 7f1fd3834000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T02:11:30.206 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:30.207+0000 7f1fd3834000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T02:11:30.280 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:30.281+0000 7f1fd3834000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T02:11:30.568 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:30.569+0000 7f1fd3834000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T02:11:31.875 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:31.875+0000 7f1fd3834000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T02:11:32.349 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:32.350+0000 7f1fd3834000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T02:11:32.429 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:32.430+0000 7f1fd3834000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T02:11:32.565 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:32.566+0000 7f1fd3834000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T02:11:33.575 INFO:tasks.ceph.mgr.x.smithi007.stderr:2024-03-29T02:11:33.575+0000 7f1f72502640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-29T02:11:35.930 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:11:36.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-29T02:11:36.037+0000 7f43c819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:36.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-29T02:11:36.042+0000 7f43c819d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:36.238 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-29T02:11:36.238 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":8,"fsid":"c741a814-a48d-480f-977e-db0279ef4ebb","created":"2024-03-29T02:11:26.745021+0000","modified":"2024-03-29T02:11:35.577689+0000","last_up_change":"2024-03-29T02:11:35.577689+0000","last_in_change":"2024-03-29T02:11:28.649749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"47c13e0f-4598-4c0e-ae31-a0fd154a39cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a699595d-34ab-499b-9e0a-85cf9c65fc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":3093812511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3093812511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":3093812511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3093812511}]},"public_addr":"172.21.15.7:6803/3093812511","cluster_addr":"172.21.15.7:6802/3093812511","heartbeat_back_addr":"172.21.15.7:6805/3093812511","heartbeat_front_addr":"172.21.15.7:6804/3093812511","state":["exists","up"]},{"osd":2,"uuid":"ffc584f3-784e-47a8-919b-4223ac892919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:11:36.250 DEBUG:teuthology.misc:1 of 3 OSDs are up 2024-03-29T02:11:42.552 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:11:42.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-29T02:11:42.664+0000 7fafeb0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:42.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-29T02:11:42.669+0000 7fafeb0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:42.862 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-29T02:11:42.862 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":12,"fsid":"c741a814-a48d-480f-977e-db0279ef4ebb","created":"2024-03-29T02:11:26.745021+0000","modified":"2024-03-29T02:11:39.591299+0000","last_up_change":"2024-03-29T02:11:36.581077+0000","last_in_change":"2024-03-29T02:11:28.649749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T02:11:36.589911+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47c13e0f-4598-4c0e-ae31-a0fd154a39cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2021624654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":2021624654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2021624654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2021624654}]},"public_addr":"172.21.15.7:6808/2021624654","cluster_addr":"172.21.15.7:6809/2021624654","heartbeat_back_addr":"172.21.15.7:6816/2021624654","heartbeat_front_addr":"172.21.15.7:6815/2021624654","state":["exists","up"]},{"osd":1,"uuid":"a699595d-34ab-499b-9e0a-85cf9c65fc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":3093812511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3093812511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":3093812511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3093812511}]},"public_addr":"172.21.15.7:6803/3093812511","cluster_addr":"172.21.15.7:6802/3093812511","heartbeat_back_addr":"172.21.15.7:6805/3093812511","heartbeat_front_addr":"172.21.15.7:6804/3093812511","state":["exists","up"]},{"osd":2,"uuid":"ffc584f3-784e-47a8-919b-4223ac892919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2950743308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2950743308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2950743308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2950743308}]},"public_addr":"172.21.15.7:6811/2950743308","cluster_addr":"172.21.15.7:6812/2950743308","heartbeat_back_addr":"172.21.15.7:6813/2950743308","heartbeat_front_addr":"172.21.15.7:6814/2950743308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:11:42.874 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-29T02:11:42.874 INFO:tasks.ceph:Creating RBD pool 2024-03-29T02:11:42.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-29T02:11:42.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:42.982+0000 7fb2c5f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:42.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:42.991+0000 7fb2c5f54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2024-03-29T02:11:43.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-29T02:11:44.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:44.063+0000 7fed6959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:44.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:44.074+0000 7fed6959d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-29T02:11:44.938 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-29T02:11:44.939 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-29T02:11:44.939 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-29T02:11:44.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-29T02:11:44.939 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-29T02:11:45.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:45.091+0000 7efe9a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:45.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:45.096+0000 7efe9a35d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:45.339 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:45.351 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":384807608},{"type":"v1","addr":"172.21.15.7:6801","nonce":384807608}]},"active_addr":"172.21.15.7:6801/384807608","active_change":"2024-03-29T02:11:32.568996+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":450059588}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2208587610}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1516438335}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1425792342}]}]} 2024-03-29T02:11:45.355 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-29T02:11:45.355 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-29T02:11:45.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:11:45.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:45.464+0000 7f7e5626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:45.469+0000 7f7e5626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:45.668 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:45.669 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"fsid":"c741a814-a48d-480f-977e-db0279ef4ebb","created":"2024-03-29T02:11:26.745021+0000","modified":"2024-03-29T02:11:44.918576+0000","last_up_change":"2024-03-29T02:11:36.581077+0000","last_in_change":"2024-03-29T02:11:28.649749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T02:11:36.589911+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-29T02:11:43.189773+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47c13e0f-4598-4c0e-ae31-a0fd154a39cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2021624654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":2021624654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2021624654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2021624654}]},"public_addr":"172.21.15.7:6808/2021624654","cluster_addr":"172.21.15.7:6809/2021624654","heartbeat_back_addr":"172.21.15.7:6816/2021624654","heartbeat_front_addr":"172.21.15.7:6815/2021624654","state":["exists","up"]},{"osd":1,"uuid":"a699595d-34ab-499b-9e0a-85cf9c65fc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":3093812511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3093812511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":3093812511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3093812511}]},"public_addr":"172.21.15.7:6803/3093812511","cluster_addr":"172.21.15.7:6802/3093812511","heartbeat_back_addr":"172.21.15.7:6805/3093812511","heartbeat_front_addr":"172.21.15.7:6804/3093812511","state":["exists","up"]},{"osd":2,"uuid":"ffc584f3-784e-47a8-919b-4223ac892919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2950743308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2950743308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2950743308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2950743308}]},"public_addr":"172.21.15.7:6811/2950743308","cluster_addr":"172.21.15.7:6812/2950743308","heartbeat_back_addr":"172.21.15.7:6813/2950743308","heartbeat_front_addr":"172.21.15.7:6814/2950743308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:11:45.681 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-29T02:11:45.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:11:45.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:45.792+0000 7fb583e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:45.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:45.796+0000 7fb583e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:45.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:45.991 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"fsid":"c741a814-a48d-480f-977e-db0279ef4ebb","created":"2024-03-29T02:11:26.745021+0000","modified":"2024-03-29T02:11:45.921344+0000","last_up_change":"2024-03-29T02:11:36.581077+0000","last_in_change":"2024-03-29T02:11:28.649749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T02:11:36.589911+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-29T02:11:43.189773+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"47c13e0f-4598-4c0e-ae31-a0fd154a39cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2021624654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":2021624654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2021624654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6815","nonce":2021624654}]},"public_addr":"172.21.15.7:6808/2021624654","cluster_addr":"172.21.15.7:6809/2021624654","heartbeat_back_addr":"172.21.15.7:6816/2021624654","heartbeat_front_addr":"172.21.15.7:6815/2021624654","state":["exists","up"]},{"osd":1,"uuid":"a699595d-34ab-499b-9e0a-85cf9c65fc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":3093812511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3093812511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":3093812511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3093812511}]},"public_addr":"172.21.15.7:6803/3093812511","cluster_addr":"172.21.15.7:6802/3093812511","heartbeat_back_addr":"172.21.15.7:6805/3093812511","heartbeat_front_addr":"172.21.15.7:6804/3093812511","state":["exists","up"]},{"osd":2,"uuid":"ffc584f3-784e-47a8-919b-4223ac892919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":2950743308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2950743308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":2950743308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2950743308}]},"public_addr":"172.21.15.7:6811/2950743308","cluster_addr":"172.21.15.7:6812/2950743308","heartbeat_back_addr":"172.21.15.7:6813/2950743308","heartbeat_front_addr":"172.21.15.7:6814/2950743308","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:11:46.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-29T02:11:46.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-29T02:11:46.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-29T02:11:46.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.120+0000 7f3967f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.122+0000 7fe6aae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.122+0000 7fada5bd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.127+0000 7f3967f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.129+0000 7fe6aae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.129+0000 7fada5bd7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.158 INFO:teuthology.orchestra.run.smithi007.stdout:34359738378 2024-03-29T02:11:46.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T02:11:46.161 INFO:teuthology.orchestra.run.smithi007.stdout:38654705673 2024-03-29T02:11:46.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:11:46.162 INFO:teuthology.orchestra.run.smithi007.stdout:38654705673 2024-03-29T02:11:46.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:11:46.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.269+0000 7f098ca60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.276+0000 7f098ca60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.317+0000 7f1ab76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.319+0000 7f15ea8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.323+0000 7f1ab76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:46.324+0000 7f15ea8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:46.480 INFO:teuthology.orchestra.run.smithi007.stdout:34359738378 2024-03-29T02:11:46.493 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738378 got 34359738378 for osd.1 2024-03-29T02:11:46.494 DEBUG:teuthology.parallel:result is None 2024-03-29T02:11:46.524 INFO:teuthology.orchestra.run.smithi007.stdout:38654705669 2024-03-29T02:11:46.526 INFO:teuthology.orchestra.run.smithi007.stdout:38654705669 2024-03-29T02:11:46.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705669 for osd.0 2024-03-29T02:11:46.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705669 for osd.2 2024-03-29T02:11:47.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:11:47.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:11:47.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:47.669+0000 7f1af8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:47.671+0000 7f177722d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:47.676+0000 7f1af8b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:47.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:47.678+0000 7f177722d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:47.893 INFO:teuthology.orchestra.run.smithi007.stdout:38654705673 2024-03-29T02:11:47.896 INFO:teuthology.orchestra.run.smithi007.stdout:38654705673 2024-03-29T02:11:47.907 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.2 2024-03-29T02:11:47.907 DEBUG:teuthology.parallel:result is None 2024-03-29T02:11:47.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705673 got 38654705673 for osd.0 2024-03-29T02:11:47.909 DEBUG:teuthology.parallel:result is None 2024-03-29T02:11:47.909 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-29T02:11:47.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T02:11:48.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:48.061+0000 7f01163e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:48.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:48.066+0000 7f01163e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:48.260 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-29T02:11:48.272 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-03-29T02:11:46.573830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80940,"kb_used_data":468,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034580,"statfs":{"total":289910292480,"available":289827409920,"internally_reserved":0,"allocated":479232,"data_stored":1264310,"data_compressed":59822,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.966918"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.045694+0000","last_change":"2024-03-29T02:11:45.045833+0000","last_active":"2024-03-29T02:11:45.045694+0000","last_peered":"2024-03-29T02:11:45.045694+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.043163+0000","last_became_peered":"2024-03-29T02:11:45.043163+0000","last_unstale":"2024-03-29T02:11:45.045694+0000","last_undegraded":"2024-03-29T02:11:45.045694+0000","last_fullsized":"2024-03-29T02:11:45.045694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.027961+0000","last_change":"2024-03-29T02:11:45.028496+0000","last_active":"2024-03-29T02:11:45.027961+0000","last_peered":"2024-03-29T02:11:45.027961+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.023672+0000","last_became_peered":"2024-03-29T02:11:45.023672+0000","last_unstale":"2024-03-29T02:11:45.027961+0000","last_undegraded":"2024-03-29T02:11:45.027961+0000","last_fullsized":"2024-03-29T02:11:45.027961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.042859+0000","last_change":"2024-03-29T02:11:45.043147+0000","last_active":"2024-03-29T02:11:45.042859+0000","last_peered":"2024-03-29T02:11:45.042859+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.038726+0000","last_became_peered":"2024-03-29T02:11:45.038726+0000","last_unstale":"2024-03-29T02:11:45.042859+0000","last_undegraded":"2024-03-29T02:11:45.042859+0000","last_fullsized":"2024-03-29T02:11:45.042859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.046660+0000","last_change":"2024-03-29T02:11:45.046780+0000","last_active":"2024-03-29T02:11:45.046660+0000","last_peered":"2024-03-29T02:11:45.046660+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.044871+0000","last_became_peered":"2024-03-29T02:11:45.044871+0000","last_unstale":"2024-03-29T02:11:45.046660+0000","last_undegraded":"2024-03-29T02:11:45.046660+0000","last_fullsized":"2024-03-29T02:11:45.046660+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.933289+0000","last_change":"2024-03-29T02:11:45.005223+0000","last_active":"2024-03-29T02:11:45.933289+0000","last_peered":"2024-03-29T02:11:45.933289+0000","last_clean":"2024-03-29T02:11:45.933289+0000","last_became_active":"2024-03-29T02:11:44.998977+0000","last_became_peered":"2024-03-29T02:11:44.998977+0000","last_unstale":"2024-03-29T02:11:45.933289+0000","last_undegraded":"2024-03-29T02:11:45.933289+0000","last_fullsized":"2024-03-29T02:11:45.933289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.010477+0000","last_change":"2024-03-29T02:11:45.010786+0000","last_active":"2024-03-29T02:11:45.010477+0000","last_peered":"2024-03-29T02:11:45.010477+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.004344+0000","last_became_peered":"2024-03-29T02:11:45.004344+0000","last_unstale":"2024-03-29T02:11:45.010477+0000","last_undegraded":"2024-03-29T02:11:45.010477+0000","last_fullsized":"2024-03-29T02:11:45.010477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:44.988484+0000","last_change":"2024-03-29T02:11:44.988913+0000","last_active":"2024-03-29T02:11:44.988484+0000","last_peered":"2024-03-29T02:11:44.988484+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:44.985770+0000","last_became_peered":"2024-03-29T02:11:44.985770+0000","last_unstale":"2024-03-29T02:11:44.988484+0000","last_undegraded":"2024-03-29T02:11:44.988484+0000","last_fullsized":"2024-03-29T02:11:44.988484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:44.987447+0000","last_change":"2024-03-29T02:11:44.987723+0000","last_active":"2024-03-29T02:11:44.987447+0000","last_peered":"2024-03-29T02:11:44.987447+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:44.985352+0000","last_became_peered":"2024-03-29T02:11:44.985352+0000","last_unstale":"2024-03-29T02:11:44.987447+0000","last_undegraded":"2024-03-29T02:11:44.987447+0000","last_fullsized":"2024-03-29T02:11:44.987447+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:44.941669+0000","last_change":"2024-03-29T02:11:38.629272+0000","last_active":"2024-03-29T02:11:44.941669+0000","last_peered":"2024-03-29T02:11:44.941669+0000","last_clean":"2024-03-29T02:11:44.941669+0000","last_became_active":"2024-03-29T02:11:38.626947+0000","last_became_peered":"2024-03-29T02:11:38.626947+0000","last_unstale":"2024-03-29T02:11:44.941669+0000","last_undegraded":"2024-03-29T02:11:44.941669+0000","last_fullsized":"2024-03-29T02:11:44.941669+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:37.584461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:37.584461+0000","last_clean_scrub_stamp":"2024-03-29T02:11:37.584461+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618226,"data_compressed":29911,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":27858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618226,"data_compressed":29911,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-29T02:11:48.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T02:11:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:48.384+0000 7f402295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:48.390+0000 7f402295d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:48.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-29T02:11:48.594 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-29T02:11:48.574148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80948,"kb_used_data":476,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034572,"statfs":{"total":289910292480,"available":289827401728,"internally_reserved":0,"allocated":487424,"data_stored":1267203,"data_compressed":59822,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.967237"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.045694+0000","last_change":"2024-03-29T02:11:45.045833+0000","last_active":"2024-03-29T02:11:45.045694+0000","last_peered":"2024-03-29T02:11:45.045694+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.043163+0000","last_became_peered":"2024-03-29T02:11:45.043163+0000","last_unstale":"2024-03-29T02:11:45.045694+0000","last_undegraded":"2024-03-29T02:11:45.045694+0000","last_fullsized":"2024-03-29T02:11:45.045694+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.027961+0000","last_change":"2024-03-29T02:11:45.028496+0000","last_active":"2024-03-29T02:11:45.027961+0000","last_peered":"2024-03-29T02:11:45.027961+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.023672+0000","last_became_peered":"2024-03-29T02:11:45.023672+0000","last_unstale":"2024-03-29T02:11:45.027961+0000","last_undegraded":"2024-03-29T02:11:45.027961+0000","last_fullsized":"2024-03-29T02:11:45.027961+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.042859+0000","last_change":"2024-03-29T02:11:45.043147+0000","last_active":"2024-03-29T02:11:45.042859+0000","last_peered":"2024-03-29T02:11:45.042859+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.038726+0000","last_became_peered":"2024-03-29T02:11:45.038726+0000","last_unstale":"2024-03-29T02:11:45.042859+0000","last_undegraded":"2024-03-29T02:11:45.042859+0000","last_fullsized":"2024-03-29T02:11:45.042859+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.046660+0000","last_change":"2024-03-29T02:11:45.046780+0000","last_active":"2024-03-29T02:11:45.046660+0000","last_peered":"2024-03-29T02:11:45.046660+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.044871+0000","last_became_peered":"2024-03-29T02:11:45.044871+0000","last_unstale":"2024-03-29T02:11:45.046660+0000","last_undegraded":"2024-03-29T02:11:45.046660+0000","last_fullsized":"2024-03-29T02:11:45.046660+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.933289+0000","last_change":"2024-03-29T02:11:45.005223+0000","last_active":"2024-03-29T02:11:45.933289+0000","last_peered":"2024-03-29T02:11:45.933289+0000","last_clean":"2024-03-29T02:11:45.933289+0000","last_became_active":"2024-03-29T02:11:44.998977+0000","last_became_peered":"2024-03-29T02:11:44.998977+0000","last_unstale":"2024-03-29T02:11:45.933289+0000","last_undegraded":"2024-03-29T02:11:45.933289+0000","last_fullsized":"2024-03-29T02:11:45.933289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:45.010477+0000","last_change":"2024-03-29T02:11:45.010786+0000","last_active":"2024-03-29T02:11:45.010477+0000","last_peered":"2024-03-29T02:11:45.010477+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:45.004344+0000","last_became_peered":"2024-03-29T02:11:45.004344+0000","last_unstale":"2024-03-29T02:11:45.010477+0000","last_undegraded":"2024-03-29T02:11:45.010477+0000","last_fullsized":"2024-03-29T02:11:45.010477+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:44.988484+0000","last_change":"2024-03-29T02:11:44.988913+0000","last_active":"2024-03-29T02:11:44.988484+0000","last_peered":"2024-03-29T02:11:44.988484+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:44.985770+0000","last_became_peered":"2024-03-29T02:11:44.985770+0000","last_unstale":"2024-03-29T02:11:44.988484+0000","last_undegraded":"2024-03-29T02:11:44.988484+0000","last_fullsized":"2024-03-29T02:11:44.988484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:44.987447+0000","last_change":"2024-03-29T02:11:44.987723+0000","last_active":"2024-03-29T02:11:44.987447+0000","last_peered":"2024-03-29T02:11:44.987447+0000","last_clean":"2024-03-29T02:11:43.914870+0000","last_became_active":"2024-03-29T02:11:44.985352+0000","last_became_peered":"2024-03-29T02:11:44.985352+0000","last_unstale":"2024-03-29T02:11:44.987447+0000","last_undegraded":"2024-03-29T02:11:44.987447+0000","last_fullsized":"2024-03-29T02:11:44.987447+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:43.914870+0000","last_clean_scrub_stamp":"2024-03-29T02:11:43.914870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-03-29T02:11:44.941669+0000","last_change":"2024-03-29T02:11:38.629272+0000","last_active":"2024-03-29T02:11:44.941669+0000","last_peered":"2024-03-29T02:11:44.941669+0000","last_clean":"2024-03-29T02:11:44.941669+0000","last_became_active":"2024-03-29T02:11:38.626947+0000","last_became_peered":"2024-03-29T02:11:38.626947+0000","last_unstale":"2024-03-29T02:11:44.941669+0000","last_undegraded":"2024-03-29T02:11:44.941669+0000","last_fullsized":"2024-03-29T02:11:44.941669+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:11:37.584461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:11:37.584461+0000","last_clean_scrub_stamp":"2024-03-29T02:11:37.584461+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":618226,"data_compressed":29911,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26940,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344900,"statfs":{"total":96636764160,"available":96609177600,"internally_reserved":0,"allocated":118784,"data_stored":27858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":184,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":188416,"data_stored":621119,"data_compressed":29911,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-29T02:11:48.595 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-29T02:11:48.595 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-29T02:11:48.595 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-29T02:11:48.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-29T02:11:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:48.706+0000 7f7a8319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:11:48.710+0000 7f7a8319d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:11:48.945 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:48.945 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-29T02:11:48.958 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-29T02:11:48.959 INFO:teuthology.run_tasks:Running task cbt... 2024-03-29T02:11:49.170 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi007.front.sepia.ceph.com', 'osds': ['ubuntu@smithi007.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi007.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi007.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi007.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-03-29T02:11:49.171 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-03-29T02:11:49.190 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:11:49.190 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-29T02:11:49.274 DEBUG:teuthology.orchestra.run.smithi007:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-03-29T02:11:49.333 INFO:teuthology.orchestra.run.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-03-29T02:11:49.721 INFO:tasks.cbt:Installing collectl 2024-03-29T02:11:49.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-03-29T02:11:49.824 INFO:teuthology.orchestra.run.smithi007.stderr:--2024-03-29 02:11:49-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-03-29T02:11:49.990 INFO:teuthology.orchestra.run.smithi007.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-03-29T02:11:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-03-29T02:11:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-29T02:11:50.331 INFO:teuthology.orchestra.run.smithi007.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmBiNmxBc7OAizLKVFoCTkBSIbmNL4TDWaZrfKk_yKqxonj8jBW57IgViU5IS5TIbYNNwKzgsFMoIDNnypt6OcOHs90A%3D%3D&use_mirror=gigenet&r= [following] 2024-03-29T02:11:50.331 INFO:teuthology.orchestra.run.smithi007.stderr:--2024-03-29 02:11:50-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmBiNmxBc7OAizLKVFoCTkBSIbmNL4TDWaZrfKk_yKqxonj8jBW57IgViU5IS5TIbYNNwKzgsFMoIDNnypt6OcOHs90A%3D%3D&use_mirror=gigenet&r= 2024-03-29T02:11:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-03-29T02:11:50.407 INFO:teuthology.orchestra.run.smithi007.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-03-29T02:11:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-29T02:11:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-03-29T02:11:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:--2024-03-29 02:11:50-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-03-29T02:11:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2024-03-29T02:11:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2024-03-29T02:11:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-29T02:11:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:Length: 620693 (606K) [application/x-gzip] 2024-03-29T02:11:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:Saving to: ‘collectl.tar.gz’ 2024-03-29T02:11:50.879 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:50.929 INFO:teuthology.orchestra.run.smithi007.stderr: 0K .......... .......... .......... .......... .......... 8% 998K 1s 2024-03-29T02:11:50.960 INFO:teuthology.orchestra.run.smithi007.stderr: 50K .......... .......... .......... .......... .......... 16% 1.59M 0s 2024-03-29T02:11:50.961 INFO:teuthology.orchestra.run.smithi007.stderr: 100K .......... .......... .......... .......... .......... 24% 46.3M 0s 2024-03-29T02:11:50.971 INFO:teuthology.orchestra.run.smithi007.stderr: 150K .......... .......... .......... .......... .......... 32% 4.46M 0s 2024-03-29T02:11:50.991 INFO:teuthology.orchestra.run.smithi007.stderr: 200K .......... .......... .......... .......... .......... 41% 2.58M 0s 2024-03-29T02:11:50.992 INFO:teuthology.orchestra.run.smithi007.stderr: 250K .......... .......... .......... .......... .......... 49% 65.6M 0s 2024-03-29T02:11:50.993 INFO:teuthology.orchestra.run.smithi007.stderr: 300K .......... .......... .......... .......... .......... 57% 69.3M 0s 2024-03-29T02:11:51.002 INFO:teuthology.orchestra.run.smithi007.stderr: 350K .......... .......... .......... .......... .......... 65% 4.73M 0s 2024-03-29T02:11:51.003 INFO:teuthology.orchestra.run.smithi007.stderr: 400K .......... .......... .......... .......... .......... 74% 79.4M 0s 2024-03-29T02:11:51.022 INFO:teuthology.orchestra.run.smithi007.stderr: 450K .......... .......... .......... .......... .......... 82% 2.63M 0s 2024-03-29T02:11:51.022 INFO:teuthology.orchestra.run.smithi007.stderr: 500K .......... .......... .......... .......... .......... 90% 76.5M 0s 2024-03-29T02:11:51.023 INFO:teuthology.orchestra.run.smithi007.stderr: 550K .......... .......... .......... .......... .......... 98% 33.7M 0s 2024-03-29T02:11:51.023 INFO:teuthology.orchestra.run.smithi007.stderr: 600K ...... 100% 11.4T=0.1s 2024-03-29T02:11:51.023 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29 02:11:51 (4.09 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-03-29T02:11:51.024 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/ 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/collectl.conf 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/ 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/SlowProc.html 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/WhySummary.html 2024-03-29T02:11:51.061 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Socket.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/colmux.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Tutorial.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Gexpr.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Import.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/style.css 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/FileNaming.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/TheMath.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Environmental.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Process.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/HiResTime.html 2024-03-29T02:11:51.062 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/slub.jpg 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Lustre.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Documentation.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Inodes.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Graphite.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/InputFiles.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Export.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Interrupts.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Startup.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Performance.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Messages.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Logging.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data-detail.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/CPUs.html 2024-03-29T02:11:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Matrix.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data-brief.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Misc.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Examples.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OpenStack.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Exceptions.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-03-29T02:11:51.064 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Playback.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Hello.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Sockets.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Network.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Infiniband.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Memory.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Disks.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/service/ 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/service/collectl.service 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/ 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl-suse 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl-debian 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl-generic 2024-03-29T02:11:51.065 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/misc.ph 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/gexpr.ph 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/statsd.ph 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/envrules.std 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/hello.ph 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/vmsum.ph 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/proctree.ph 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/README-WINDOWS 2024-03-29T02:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/formatit.ph 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/colmux 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/INSTALL 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/graphite.ph 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/RELEASE-collectl 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/vmstat.ph 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/README 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/man1/ 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/man1/collectl.1 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/man1/colmux.1 2024-03-29T02:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/lexpr.ph 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/client.pl 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/collectl 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/COPYING 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/vnet.ph 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/GPL 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/UNINSTALL 2024-03-29T02:11:51.069 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/ARTISTIC 2024-03-29T02:11:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:./INSTALL: line 90: chkconfig: command not found 2024-03-29T02:11:51.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-03-29T02:11:51.708 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:19 ago on Fri 29 Mar 2024 02:08:32 AM UTC. 2024-03-29T02:11:51.810 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-03-29T02:11:51.813 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-03-29T02:11:51.873 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: librbd-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 23 k 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: perf x86_64 5.14.0-434.el9 CentOS-AppStream 2.2 M 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: libradospp-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 30 k 2024-03-29T02:11:51.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.3 M 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 11 M 2024-03-29T02:11:51.875 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-29T02:11:52.515 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): perf-5.14.0-434.el9.x86_64.rpm 7.9 MB/s | 2.2 MB 00:00 2024-03-29T02:11:52.540 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): librbd-devel-19.0.0-2473.g785fc5e8.el9.x 77 kB/s | 23 kB 00:00 2024-03-29T02:11:52.565 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): libradospp-devel-19.0.0-2473.g785fc5e8.e 94 kB/s | 30 kB 00:00 2024-03-29T02:11:52.758 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 60 kB/s | 13 kB 00:00 2024-03-29T02:11:52.883 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 323 kB/s | 118 kB 00:00 2024-03-29T02:11:52.886 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:11:52.886 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.3 MB/s | 2.3 MB 00:01 2024-03-29T02:11:52.887 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-29T02:11:52.907 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-29T02:11:52.907 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-29T02:11:53.030 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-29T02:11:53.031 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-29T02:11:53.436 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-29T02:11:53.502 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-03-29T02:11:53.533 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-03-29T02:11:53.563 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/5 2024-03-29T02:11:53.892 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/5 2024-03-29T02:11:53.981 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : perf-5.14.0-434.el9.x86_64 5/5 2024-03-29T02:11:54.986 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: perf-5.14.0-434.el9.x86_64 5/5 2024-03-29T02:11:54.986 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/5 2024-03-29T02:11:54.986 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/5 2024-03-29T02:11:54.986 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : perf-5.14.0-434.el9.x86_64 3/5 2024-03-29T02:11:54.986 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-03-29T02:11:55.418 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh-2.34-7.el9.x86_64 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: perf-5.14.0-434.el9.x86_64 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:55.419 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-29T02:11:55.776 INFO:tasks.cbt:benchmark: librbdfio 2024-03-29T02:11:55.777 DEBUG:teuthology.orchestra.run.smithi007:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-03-29T02:11:55.792 INFO:teuthology.orchestra.run.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-03-29T02:11:58.109 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:Operating system Linux 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:CPU x86_64 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:Big endian no 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:Compiler gcc 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:Cross compile no 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:11:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:Static build no 2024-03-29T02:11:58.772 INFO:teuthology.orchestra.run.smithi007.stdout:Wordsize 64 2024-03-29T02:11:58.779 INFO:teuthology.orchestra.run.smithi007.stdout:zlib no 2024-03-29T02:11:58.853 INFO:teuthology.orchestra.run.smithi007.stdout:fcntl(F_FULLFSYNC) no 2024-03-29T02:11:58.867 INFO:teuthology.orchestra.run.smithi007.stdout:Linux AIO support no 2024-03-29T02:11:58.867 INFO:teuthology.orchestra.run.smithi007.stdout:Linux AIO support rw flags no 2024-03-29T02:11:58.924 INFO:teuthology.orchestra.run.smithi007.stdout:POSIX AIO support yes 2024-03-29T02:11:58.924 INFO:teuthology.orchestra.run.smithi007.stdout:POSIX AIO support needs -lrt no 2024-03-29T02:11:58.956 INFO:teuthology.orchestra.run.smithi007.stdout:POSIX AIO fsync yes 2024-03-29T02:11:58.986 INFO:teuthology.orchestra.run.smithi007.stdout:POSIX pshared support yes 2024-03-29T02:11:59.030 INFO:teuthology.orchestra.run.smithi007.stdout:pthread_condattr_setclock() yes 2024-03-29T02:11:59.067 INFO:teuthology.orchestra.run.smithi007.stdout:pthread_sigmask() yes 2024-03-29T02:11:59.103 INFO:teuthology.orchestra.run.smithi007.stdout:pthread_getaffinity_np() yes 2024-03-29T02:11:59.112 INFO:teuthology.orchestra.run.smithi007.stdout:Solaris AIO support no 2024-03-29T02:11:59.165 INFO:teuthology.orchestra.run.smithi007.stdout:__sync_fetch_and_add yes 2024-03-29T02:11:59.219 INFO:teuthology.orchestra.run.smithi007.stdout:__sync_synchronize yes 2024-03-29T02:11:59.248 INFO:teuthology.orchestra.run.smithi007.stdout:__sync_val_compare_and_swap yes 2024-03-29T02:11:59.255 INFO:teuthology.orchestra.run.smithi007.stdout:libverbs no 2024-03-29T02:11:59.318 INFO:teuthology.orchestra.run.smithi007.stdout:rdmacm no 2024-03-29T02:11:59.326 INFO:teuthology.orchestra.run.smithi007.stdout:librpma no 2024-03-29T02:11:59.335 INFO:teuthology.orchestra.run.smithi007.stdout:libprotobuf_c no 2024-03-29T02:11:59.365 INFO:teuthology.orchestra.run.smithi007.stdout:asprintf() yes 2024-03-29T02:11:59.396 INFO:teuthology.orchestra.run.smithi007.stdout:vasprintf() yes 2024-03-29T02:11:59.427 INFO:teuthology.orchestra.run.smithi007.stdout:Linux fallocate yes 2024-03-29T02:11:59.458 INFO:teuthology.orchestra.run.smithi007.stdout:POSIX fadvise yes 2024-03-29T02:11:59.490 INFO:teuthology.orchestra.run.smithi007.stdout:POSIX fallocate yes 2024-03-29T02:11:59.519 INFO:teuthology.orchestra.run.smithi007.stdout:sched_setaffinity(3 arg) yes 2024-03-29T02:11:59.519 INFO:teuthology.orchestra.run.smithi007.stdout:sched_setaffinity(2 arg) no 2024-03-29T02:11:59.550 INFO:teuthology.orchestra.run.smithi007.stdout:clock_gettime yes 2024-03-29T02:11:59.582 INFO:teuthology.orchestra.run.smithi007.stdout:CLOCK_MONOTONIC yes 2024-03-29T02:11:59.628 INFO:teuthology.orchestra.run.smithi007.stdout:clockid_t yes 2024-03-29T02:11:59.665 INFO:teuthology.orchestra.run.smithi007.stdout:gettimeofday yes 2024-03-29T02:11:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:fdatasync yes 2024-03-29T02:11:59.727 INFO:teuthology.orchestra.run.smithi007.stdout:pipe() yes 2024-03-29T02:11:59.759 INFO:teuthology.orchestra.run.smithi007.stdout:pipe2() yes 2024-03-29T02:11:59.788 INFO:teuthology.orchestra.run.smithi007.stdout:pread() yes 2024-03-29T02:11:59.862 INFO:teuthology.orchestra.run.smithi007.stdout:sync_file_range yes 2024-03-29T02:11:59.869 INFO:teuthology.orchestra.run.smithi007.stdout:ASharedMemory_create no 2024-03-29T02:11:59.896 INFO:teuthology.orchestra.run.smithi007.stdout:EXT4 move extent yes 2024-03-29T02:11:59.927 INFO:teuthology.orchestra.run.smithi007.stdout:Linux splice(2) yes 2024-03-29T02:11:59.933 INFO:teuthology.orchestra.run.smithi007.stdout:libnuma no 2024-03-29T02:11:59.963 INFO:teuthology.orchestra.run.smithi007.stdout:strsep yes 2024-03-29T02:11:59.993 INFO:teuthology.orchestra.run.smithi007.stdout:strcasestr yes 2024-03-29T02:12:00.006 INFO:teuthology.orchestra.run.smithi007.stdout:strlcat no 2024-03-29T02:12:00.040 INFO:teuthology.orchestra.run.smithi007.stdout:getopt_long_only() yes 2024-03-29T02:12:00.120 INFO:teuthology.orchestra.run.smithi007.stdout:inet_aton yes 2024-03-29T02:12:00.151 INFO:teuthology.orchestra.run.smithi007.stdout:socklen_t yes 2024-03-29T02:12:00.182 INFO:teuthology.orchestra.run.smithi007.stdout:__thread yes 2024-03-29T02:12:00.216 INFO:teuthology.orchestra.run.smithi007.stdout:RUSAGE_THREAD yes 2024-03-29T02:12:00.245 INFO:teuthology.orchestra.run.smithi007.stdout:SCHED_IDLE yes 2024-03-29T02:12:00.280 INFO:teuthology.orchestra.run.smithi007.stdout:TCP_NODELAY yes 2024-03-29T02:12:00.327 INFO:teuthology.orchestra.run.smithi007.stdout:vsock yes 2024-03-29T02:12:00.362 INFO:teuthology.orchestra.run.smithi007.stdout:Net engine window_size yes 2024-03-29T02:12:00.396 INFO:teuthology.orchestra.run.smithi007.stdout:TCP_MAXSEG yes 2024-03-29T02:12:00.425 INFO:teuthology.orchestra.run.smithi007.stdout:RLIMIT_MEMLOCK yes 2024-03-29T02:12:00.458 INFO:teuthology.orchestra.run.smithi007.stdout:pwritev/preadv yes 2024-03-29T02:12:00.491 INFO:teuthology.orchestra.run.smithi007.stdout:pwritev2/preadv2 yes 2024-03-29T02:12:00.538 INFO:teuthology.orchestra.run.smithi007.stdout:IPv6 helpers yes 2024-03-29T02:12:00.553 INFO:teuthology.orchestra.run.smithi007.stdout:http engine no 2024-03-29T02:12:00.697 INFO:teuthology.orchestra.run.smithi007.stdout:Rados engine yes 2024-03-29T02:12:00.850 INFO:teuthology.orchestra.run.smithi007.stdout:Rados Block Device engine yes 2024-03-29T02:12:01.016 INFO:teuthology.orchestra.run.smithi007.stdout:rbd_poll yes 2024-03-29T02:12:01.174 INFO:teuthology.orchestra.run.smithi007.stdout:rbd_invalidate_cache yes 2024-03-29T02:12:01.202 INFO:teuthology.orchestra.run.smithi007.stdout:setvbuf yes 2024-03-29T02:12:01.209 INFO:teuthology.orchestra.run.smithi007.stdout:Gluster API engine no 2024-03-29T02:12:01.224 INFO:teuthology.orchestra.run.smithi007.stdout:s390_z196_facilities no 2024-03-29T02:12:01.224 INFO:teuthology.orchestra.run.smithi007.stdout:HDFS engine no 2024-03-29T02:12:01.265 INFO:teuthology.orchestra.run.smithi007.stdout:MTD yes 2024-03-29T02:12:01.272 INFO:teuthology.orchestra.run.smithi007.stdout:libpmem no 2024-03-29T02:12:01.272 INFO:teuthology.orchestra.run.smithi007.stdout:libpmem1_5 no 2024-03-29T02:12:01.279 INFO:teuthology.orchestra.run.smithi007.stdout:libpmem2 no 2024-03-29T02:12:01.280 INFO:teuthology.orchestra.run.smithi007.stdout:PMDK dev-dax engine no 2024-03-29T02:12:01.280 INFO:teuthology.orchestra.run.smithi007.stdout:PMDK libpmem engine no 2024-03-29T02:12:01.287 INFO:teuthology.orchestra.run.smithi007.stdout:DDN's Infinite Memory Engine no 2024-03-29T02:12:01.287 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi engine no 2024-03-29T02:12:01.287 INFO:teuthology.orchestra.run.smithi007.stdout:NBD engine no 2024-03-29T02:12:01.296 INFO:teuthology.orchestra.run.smithi007.stdout:DAOS File System (dfs) Engine no 2024-03-29T02:12:01.305 INFO:teuthology.orchestra.run.smithi007.stdout:NFS engine no 2024-03-29T02:12:01.305 INFO:teuthology.orchestra.run.smithi007.stdout:lex/yacc for arithmetic no 2024-03-29T02:12:01.344 INFO:teuthology.orchestra.run.smithi007.stdout:getmntent yes 2024-03-29T02:12:01.372 INFO:teuthology.orchestra.run.smithi007.stdout:getmntinfo no 2024-03-29T02:12:01.433 INFO:teuthology.orchestra.run.smithi007.stdout:Static Assert yes 2024-03-29T02:12:01.460 INFO:teuthology.orchestra.run.smithi007.stdout:bool yes 2024-03-29T02:12:01.494 INFO:teuthology.orchestra.run.smithi007.stdout:strndup yes 2024-03-29T02:12:01.502 INFO:teuthology.orchestra.run.smithi007.stdout:Valgrind headers no 2024-03-29T02:12:01.537 INFO:teuthology.orchestra.run.smithi007.stdout:Zoned block device support yes 2024-03-29T02:12:01.564 INFO:teuthology.orchestra.run.smithi007.stdout:Zoned block device capacity yes 2024-03-29T02:12:01.572 INFO:teuthology.orchestra.run.smithi007.stdout:libzbc engine no 2024-03-29T02:12:01.600 INFO:teuthology.orchestra.run.smithi007.stdout:NVMe uring command support yes 2024-03-29T02:12:01.615 INFO:teuthology.orchestra.run.smithi007.stdout:xnvme engine no 2024-03-29T02:12:01.622 INFO:teuthology.orchestra.run.smithi007.stdout:isal no 2024-03-29T02:12:01.636 INFO:teuthology.orchestra.run.smithi007.stdout:libblkio engine no 2024-03-29T02:12:01.636 INFO:teuthology.orchestra.run.smithi007.stdout:march_armv8_a_crc_crypto no 2024-03-29T02:12:01.636 INFO:teuthology.orchestra.run.smithi007.stdout:cuda no 2024-03-29T02:12:01.636 INFO:teuthology.orchestra.run.smithi007.stdout:libcufile no 2024-03-29T02:12:01.664 INFO:teuthology.orchestra.run.smithi007.stdout:Build march=native yes 2024-03-29T02:12:01.672 INFO:teuthology.orchestra.run.smithi007.stdout:CUnit no 2024-03-29T02:12:01.700 INFO:teuthology.orchestra.run.smithi007.stdout:__kernel_rwf_t yes 2024-03-29T02:12:01.727 INFO:teuthology.orchestra.run.smithi007.stdout:-Wimplicit-fallthrough=2 yes 2024-03-29T02:12:01.759 INFO:teuthology.orchestra.run.smithi007.stdout:-Wno-stringop-truncation yes 2024-03-29T02:12:01.790 INFO:teuthology.orchestra.run.smithi007.stdout:MADV_HUGEPAGE yes 2024-03-29T02:12:01.820 INFO:teuthology.orchestra.run.smithi007.stdout:gettid yes 2024-03-29T02:12:01.855 INFO:teuthology.orchestra.run.smithi007.stdout:statx(2)/libc yes 2024-03-29T02:12:01.888 INFO:teuthology.orchestra.run.smithi007.stdout:statx(2)/syscall yes 2024-03-29T02:12:01.890 INFO:teuthology.orchestra.run.smithi007.stdout:Windows PDB generation no 2024-03-29T02:12:01.920 INFO:teuthology.orchestra.run.smithi007.stdout:timerfd_create yes 2024-03-29T02:12:01.923 INFO:teuthology.orchestra.run.smithi007.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-03-29T02:12:01.923 INFO:teuthology.orchestra.run.smithi007.stdout:Lib-based ioengines dynamic no 2024-03-29T02:12:01.985 INFO:teuthology.orchestra.run.smithi007.stdout:TCMalloc support yes 2024-03-29T02:12:01.986 INFO:teuthology.orchestra.run.smithi007.stdout:seed_buckets 4 2024-03-29T02:12:02.066 INFO:teuthology.orchestra.run.smithi007.stderr:FIO_VERSION = fio-3.37 2024-03-29T02:12:02.097 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc16.o 2024-03-29T02:12:02.184 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc32.o 2024-03-29T02:12:02.219 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc32c-arm64.o 2024-03-29T02:12:02.289 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc32c-intel.o 2024-03-29T02:12:02.332 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc32c.o 2024-03-29T02:12:02.371 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc64.o 2024-03-29T02:12:02.409 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crc7.o 2024-03-29T02:12:02.442 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/crct10dif_common.o 2024-03-29T02:12:02.482 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/fnv.o 2024-03-29T02:12:02.521 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/md5.o 2024-03-29T02:12:02.626 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/murmur3.o 2024-03-29T02:12:02.669 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/sha1.o 2024-03-29T02:12:02.868 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/sha256.o 2024-03-29T02:12:03.170 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/sha3.o 2024-03-29T02:12:03.322 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/sha512.o 2024-03-29T02:12:03.431 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/test.o 2024-03-29T02:12:03.599 INFO:teuthology.orchestra.run.smithi007.stdout: CC crc/xxhash.o 2024-03-29T02:12:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/axmap.o 2024-03-29T02:12:03.826 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/bloom.o 2024-03-29T02:12:03.908 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/flist_sort.o 2024-03-29T02:12:03.979 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/gauss.o 2024-03-29T02:12:04.076 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/getrusage.o 2024-03-29T02:12:04.110 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/hweight.o 2024-03-29T02:12:04.148 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/ieee754.o 2024-03-29T02:12:04.198 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/lfsr.o 2024-03-29T02:12:04.304 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/memalign.o 2024-03-29T02:12:04.347 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/memcpy.o 2024-03-29T02:12:04.495 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/mountcheck.o 2024-03-29T02:12:04.535 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/num2str.o 2024-03-29T02:12:04.603 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/output_buffer.o 2024-03-29T02:12:04.649 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/pattern.o 2024-03-29T02:12:04.799 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/prio_tree.o 2024-03-29T02:12:04.925 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/rand.o 2024-03-29T02:12:05.120 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/rbtree.o 2024-03-29T02:12:05.247 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/strntol.o 2024-03-29T02:12:05.294 INFO:teuthology.orchestra.run.smithi007.stdout: CC lib/zipf.o 2024-03-29T02:12:05.415 INFO:teuthology.orchestra.run.smithi007.stdout: CC gettime.o 2024-03-29T02:12:05.718 INFO:teuthology.orchestra.run.smithi007.stdout: CC ioengines.o 2024-03-29T02:12:05.992 INFO:teuthology.orchestra.run.smithi007.stdout: CC init.o 2024-03-29T02:12:06.774 INFO:teuthology.orchestra.run.smithi007.stdout: CC stat.o 2024-03-29T02:12:08.714 INFO:teuthology.orchestra.run.smithi007.stdout: CC log.o 2024-03-29T02:12:08.837 INFO:teuthology.orchestra.run.smithi007.stdout: CC time.o 2024-03-29T02:12:08.962 INFO:teuthology.orchestra.run.smithi007.stdout: CC filesetup.o 2024-03-29T02:12:09.697 INFO:teuthology.orchestra.run.smithi007.stdout: CC eta.o 2024-03-29T02:12:10.027 INFO:teuthology.orchestra.run.smithi007.stdout: CC verify.o 2024-03-29T02:12:10.742 INFO:teuthology.orchestra.run.smithi007.stdout: CC memory.o 2024-03-29T02:12:10.914 INFO:teuthology.orchestra.run.smithi007.stdout: CC io_u.o 2024-03-29T02:12:11.726 INFO:teuthology.orchestra.run.smithi007.stdout: CC parse.o 2024-03-29T02:12:12.193 INFO:teuthology.orchestra.run.smithi007.stdout: CC fio_sem.o 2024-03-29T02:12:12.297 INFO:teuthology.orchestra.run.smithi007.stdout: CC rwlock.o 2024-03-29T02:12:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: CC pshared.o 2024-03-29T02:12:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: CC options.o 2024-03-29T02:12:13.262 INFO:teuthology.orchestra.run.smithi007.stdout: CC smalloc.o 2024-03-29T02:12:13.450 INFO:teuthology.orchestra.run.smithi007.stdout: CC filehash.o 2024-03-29T02:12:13.606 INFO:teuthology.orchestra.run.smithi007.stdout: CC profile.o 2024-03-29T02:12:13.723 INFO:teuthology.orchestra.run.smithi007.stdout: CC debug.o 2024-03-29T02:12:13.764 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/cpu.o 2024-03-29T02:12:13.902 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/mmap.o 2024-03-29T02:12:14.069 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/sync.o 2024-03-29T02:12:14.305 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/null.o 2024-03-29T02:12:14.424 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/net.o 2024-03-29T02:12:14.895 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/ftruncate.o 2024-03-29T02:12:14.993 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/fileoperations.o 2024-03-29T02:12:15.152 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/exec.o 2024-03-29T02:12:15.309 INFO:teuthology.orchestra.run.smithi007.stdout: CC server.o 2024-03-29T02:12:16.077 INFO:teuthology.orchestra.run.smithi007.stdout: CC client.o 2024-03-29T02:12:16.639 INFO:teuthology.orchestra.run.smithi007.stdout: CC iolog.o 2024-03-29T02:12:17.103 INFO:teuthology.orchestra.run.smithi007.stdout: CC backend.o 2024-03-29T02:12:17.973 INFO:teuthology.orchestra.run.smithi007.stdout: CC libfio.o 2024-03-29T02:12:18.155 INFO:teuthology.orchestra.run.smithi007.stdout: CC flow.o 2024-03-29T02:12:18.273 INFO:teuthology.orchestra.run.smithi007.stdout: CC cconv.o 2024-03-29T02:12:18.639 INFO:teuthology.orchestra.run.smithi007.stdout: CC gettime-thread.o 2024-03-29T02:12:18.743 INFO:teuthology.orchestra.run.smithi007.stdout: CC helpers.o 2024-03-29T02:12:18.805 INFO:teuthology.orchestra.run.smithi007.stdout: CC json.o 2024-03-29T02:12:18.990 INFO:teuthology.orchestra.run.smithi007.stdout: CC idletime.o 2024-03-29T02:12:19.223 INFO:teuthology.orchestra.run.smithi007.stdout: CC td_error.o 2024-03-29T02:12:19.321 INFO:teuthology.orchestra.run.smithi007.stdout: CC profiles/tiobench.o 2024-03-29T02:12:19.418 INFO:teuthology.orchestra.run.smithi007.stdout: CC profiles/act.o 2024-03-29T02:12:19.586 INFO:teuthology.orchestra.run.smithi007.stdout: CC io_u_queue.o 2024-03-29T02:12:19.634 INFO:teuthology.orchestra.run.smithi007.stdout: CC filelock.o 2024-03-29T02:12:19.762 INFO:teuthology.orchestra.run.smithi007.stdout: CC workqueue.o 2024-03-29T02:12:19.945 INFO:teuthology.orchestra.run.smithi007.stdout: CC rate-submit.o 2024-03-29T02:12:20.094 INFO:teuthology.orchestra.run.smithi007.stdout: CC optgroup.o 2024-03-29T02:12:20.140 INFO:teuthology.orchestra.run.smithi007.stdout: CC helper_thread.o 2024-03-29T02:12:20.324 INFO:teuthology.orchestra.run.smithi007.stdout: CC steadystate.o 2024-03-29T02:12:20.569 INFO:teuthology.orchestra.run.smithi007.stdout: CC zone-dist.o 2024-03-29T02:12:20.706 INFO:teuthology.orchestra.run.smithi007.stdout: CC zbd.o 2024-03-29T02:12:21.430 INFO:teuthology.orchestra.run.smithi007.stdout: CC dedupe.o 2024-03-29T02:12:21.574 INFO:teuthology.orchestra.run.smithi007.stdout: CC fdp.o 2024-03-29T02:12:21.718 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/posixaio.o 2024-03-29T02:12:21.855 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/falloc.o 2024-03-29T02:12:21.968 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/e4defrag.o 2024-03-29T02:12:22.102 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/splice.o 2024-03-29T02:12:22.294 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/asprintf.o 2024-03-29T02:12:22.330 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/strlcat.o 2024-03-29T02:12:22.374 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/mtd.o 2024-03-29T02:12:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/libmtd.o 2024-03-29T02:12:22.932 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/libmtd_legacy.o 2024-03-29T02:12:23.066 INFO:teuthology.orchestra.run.smithi007.stdout: CC diskutil.o 2024-03-29T02:12:23.279 INFO:teuthology.orchestra.run.smithi007.stdout: CC fifo.o 2024-03-29T02:12:23.334 INFO:teuthology.orchestra.run.smithi007.stdout: CC blktrace.o 2024-03-29T02:12:23.677 INFO:teuthology.orchestra.run.smithi007.stdout: CC cgroup.o 2024-03-29T02:12:23.830 INFO:teuthology.orchestra.run.smithi007.stdout: CC trim.o 2024-03-29T02:12:23.947 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/sg.o 2024-03-29T02:12:24.344 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/linux-dev-lookup.o 2024-03-29T02:12:24.401 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/io_uring.o 2024-03-29T02:12:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/nvme.o 2024-03-29T02:12:25.124 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/linux-blkzoned.o 2024-03-29T02:12:25.279 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/cmdprio.o 2024-03-29T02:12:25.523 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/rados.o 2024-03-29T02:12:25.700 INFO:teuthology.orchestra.run.smithi007.stdout: CC engines/rbd.o 2024-03-29T02:12:25.889 INFO:teuthology.orchestra.run.smithi007.stdout: CC fio.o 2024-03-29T02:12:25.977 INFO:teuthology.orchestra.run.smithi007.stdout: LINK fio 2024-03-29T02:12:26.233 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/log.o 2024-03-29T02:12:26.275 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/genzipf.o 2024-03-29T02:12:26.417 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/strcasestr.o 2024-03-29T02:12:26.441 INFO:teuthology.orchestra.run.smithi007.stdout: CC oslib/strndup.o 2024-03-29T02:12:26.463 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/fio-genzipf 2024-03-29T02:12:26.608 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/btrace2fio.o 2024-03-29T02:12:27.043 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/fio-btrace2fio 2024-03-29T02:12:27.185 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/verify-state.o 2024-03-29T02:12:27.284 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/debug.o 2024-03-29T02:12:27.318 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/fio-verify-state 2024-03-29T02:12:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/stest.o 2024-03-29T02:12:27.530 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/arch.o 2024-03-29T02:12:27.559 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/stest 2024-03-29T02:12:27.700 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/ieee754.o 2024-03-29T02:12:27.739 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/ieee754 2024-03-29T02:12:27.880 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/axmap.o 2024-03-29T02:12:27.981 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/axmap 2024-03-29T02:12:28.123 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/lfsr-test.o 2024-03-29T02:12:28.219 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/lfsr-test 2024-03-29T02:12:28.360 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/gen-rand.o 2024-03-29T02:12:28.460 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/gen-rand 2024-03-29T02:12:28.604 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/memlock.o 2024-03-29T02:12:28.665 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/memlock 2024-03-29T02:12:28.805 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/read-to-pipe-async.o 2024-03-29T02:12:28.997 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/read-to-pipe-async 2024-03-29T02:12:29.138 INFO:teuthology.orchestra.run.smithi007.stdout: CC t/io_uring.o 2024-03-29T02:12:29.693 INFO:teuthology.orchestra.run.smithi007.stdout: LINK t/io_uring 2024-03-29T02:12:29.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-03-29T02:12:29.866 DEBUG:teuthology.orchestra.run.smithi007:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - Settings.cluster: 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: 'clients': ['ubuntu@smithi007.front.sepia.ceph.com'], 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: 'head': 'ubuntu@smithi007.front.sepia.ceph.com', 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: 'is_teuthology': True, 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: 'iterations': 1, 2024-03-29T02:12:30.118 INFO:teuthology.orchestra.run.smithi007.stderr: 'mons': ['ubuntu@smithi007.front.sepia.ceph.com'], 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'osds': ['ubuntu@smithi007.front.sepia.ceph.com'], 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'osds_per_node': 3, 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'pid_dir': '/var/run/ceph', 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'rebuild_every_test': False, 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'rgws': {'ubuntu@smithi007.front.sepia.ceph.com': None}, 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'tmp_dir': '/tmp/cbt', 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'use_existing': True, 2024-03-29T02:12:30.119 INFO:teuthology.orchestra.run.smithi007.stderr: 'user': 'ubuntu'} 2024-03-29T02:12:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread; fi 2024-03-29T02:12:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.122 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-03-29T02:12:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-03-29T02:12:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2024-03-29T02:12:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:12:30.129 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.130 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.130 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2024-03-29T02:12:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:12:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:12:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:02:12:30 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:13:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.182 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:13:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:13:30.279 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.280 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.280 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2024-03-29T02:13:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi007.front.sepia.ceph.com ; done 2024-03-29T02:13:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2024-03-29T02:13:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:13:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.305 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2024-03-29T02:13:30.308 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.308 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:13:30.308 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:13:30.308 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.308 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.308 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-03-29T02:13:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-03-29T02:13:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:31 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:31 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:31.704 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-03-29T02:13:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:32 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:32 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-03-29T02:13:33.713 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:33 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:33 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:33.714 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-03-29T02:13:34.712 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:34 - INFO - cbt - Checking Health after pool creation. 2024-03-29T02:13:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:39 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:39 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-29T02:13:40.064 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:40 - INFO - cbt - Final Pool Health Check. 2024-03-29T02:13:45.069 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:45.070 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:45.070 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-29T02:13:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:45.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-03-29T02:13:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi007.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-03-29T02:13:47.022 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.023 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.023 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi007.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-03-29T02:13:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:13:47.156 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.157 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.157 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:13:47.197 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - INFO - cbt - Attempting to populating fio files... 2024-03-29T02:13:47.197 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.197 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.197 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-03-29T02:13:47.198 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.198 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:13:47.198 INFO:teuthology.orchestra.run.smithi007.stderr:02:13:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-03-29T02:15:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c. 2024-03-29T02:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - INFO - cbt - Setting OSD Read Ahead to: 4096 2024-03-29T02:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2024-03-29T02:15:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-03-29T02:15:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-03-29T02:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-03-29T02:15:52.099 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:15:52.099 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-03-29T02:15:52.839 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:15:52.840 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:15:52.840 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-03-29T02:15:52.991 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:15:52.991 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:15:52.992 INFO:teuthology.orchestra.run.smithi007.stderr:02:15:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2024-03-29T02:16:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2024-03-29T02:16:19.410 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.411 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl 2024-03-29T02:16:19.413 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.413 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:16:19.414 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.414 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.415 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - INFO - cbt - Running rbd fio randread test. 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.418 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 2024-03-29T02:16:19.419 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2024-03-29T02:16:19.419 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.419 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:16:19.419 INFO:teuthology.orchestra.run.smithi007.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 2024-03-29T02:17:20.452 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.452 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.452 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:17:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:17:20.499 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.499 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2024-03-29T02:17:20.501 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.501 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2024-03-29T02:17:20.502 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.502 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.502 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2024-03-29T02:17:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.850 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.850 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2024-03-29T02:17:20.867 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.867 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi007.front.sepia.ceph.com ; done 2024-03-29T02:17:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - INFO - cbt - Convert results to json format. 2024-03-29T02:17:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:02:17:20 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.943 INFO:teuthology.orchestra.run.smithi007.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-03-29T02:17:20.943 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007.front.sepia.ceph.com: smithi007.front.sepia.ceph.com 2024-03-29T02:17:20.943 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-29T02:17:20.943 INFO:teuthology.orchestra.run.smithi007.stdout:['smithi007.front.sepia.ceph.com'] 2024-03-29T02:17:20.971 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-03-29T02:17:20.981 DEBUG:teuthology.orchestra.run.smithi007:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-03-29T02:17:21.151 DEBUG:teuthology.orchestra.run.smithi007:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-03-29T02:17:21.214 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-03-29T02:17:21.214 INFO:tasks.cbt_performance:reading cbt preferences 2024-03-29T02:17:21.214 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-03-29T02:17:21.288 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi007.front.sepia.ceph.com 2024-03-29T02:17:21.288 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi007.front.sepia.ceph.com 2024-03-29T02:17:21.290 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:17:21.290 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi007.front.sepia.ceph.com of=/dev/stdout 2024-03-29T02:17:21.307 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-29T02:17:21.307 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2024-03-29T02:17:21.361 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-29T02:17:21.362 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-29T02:17:21.362 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-29T02:17:21.362 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi007.front.sepia.ceph.com of=/dev/stdout 2024-03-29T02:17:21.418 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-29T02:17:21.419 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2024-03-29T02:17:21.473 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-29T02:17:21.473 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-29T02:17:21.486 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-29T02:17:21.494 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-29T02:17:21.494 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T02:17:21.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-29T02:17:21.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:17:21.622+0000 7f9d2ddf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:17:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-29T02:17:21.631+0000 7f9d2ddf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:17:21.844 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-29T02:17:21.844 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T02:17:21.844 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T02:17:21.844 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:17:26.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-29T02:17:27.946 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T02:17:27.946 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T02:17:27.946 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:17:31.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-29T02:17:31.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-29T02:17:34.048 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T02:17:34.049 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T02:17:34.049 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:17:37.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-29T02:17:37.075 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-29T02:17:40.151 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T02:17:40.151 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T02:17:40.151 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T02:17:40.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:17:40.184 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T02:17:40.184 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T02:17:40.184 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T02:17:40.184 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:17:40.237 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T02:17:40.237 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T02:17:40.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-29T02:17:40.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:40.334 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T02:17:40.584 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:40.584 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T02:17:40.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:17:40.696 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T02:17:40.753 INFO:tasks.ceph:Archiving mon data... 2024-03-29T02:17:40.754 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041/data/mon.a.tgz 2024-03-29T02:17:40.754 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-29T02:17:40.780 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.KO3FJegXiK 2024-03-29T02:17:40.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KO3FJegXiK 2024-03-29T02:17:40.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.KO3FJegXiK 2024-03-29T02:17:41.046 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.KO3FJegXiK is 200KB 2024-03-29T02:17:41.070 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.KO3FJegXiK 2024-03-29T02:17:41.084 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T02:17:41.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T02:17:41.218 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T02:17:41.218 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T02:17:41.219 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041/remote/smithi007/crash 2024-03-29T02:17:41.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T02:17:41.283 INFO:tasks.ceph:Compressing logs... 2024-03-29T02:17:41.284 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T02:17:41.385 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35401.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35283.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:gzip/var/log/ceph/ceph-client.admin.35401.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35233.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35818.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30198.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30791.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35818.log.gz 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30008.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30118.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35306.log 2024-03-29T02:17:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31243.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30907.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35532.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30949.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35532.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35532.log.gz 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31157.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31155.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35353.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31155.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29920.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31282.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.29920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35531.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30991.log 2024-03-29T02:17:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35531.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28549.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35531.log.gz 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31453.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.28549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35206.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35142.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31156.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35370.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35142.log: /var/log/ceph/ceph-client.admin.31156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31075.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30851.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31283.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31388.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31283.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31495.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35329.log 2024-03-29T02:17:41.388 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35484.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35402.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35484.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35484.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30158.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29920.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30008.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30356.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35206.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35233.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31033.log 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35306.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.35353.log.gz 2024-03-29T02:17:41.389 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35401.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30356.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31537.log 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35142.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.35370.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30386.log 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.31537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35258.log 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.30386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31389.log 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35329.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35402.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35258.log: /var/log/ceph/ceph-client.admin.31389.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35258.log.gz 2024-03-29T02:17:41.390 INFO:teuthology.orchestra.run.smithi007.stderr: 86.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T02:17:41.477 INFO:teuthology.orchestra.run.smithi007.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T02:17:41.549 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30791.log.gz 2024-03-29T02:17:41.550 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30851.log.gz 2024-03-29T02:17:41.550 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30949.log.gz 2024-03-29T02:17:41.550 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30991.log.gz 2024-03-29T02:17:41.550 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30907.log.gz 2024-03-29T02:17:41.550 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31033.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31075.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31155.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31157.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31156.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31243.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31282.log.gz 2024-03-29T02:17:41.551 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31283.log.gz 2024-03-29T02:17:41.552 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31389.log.gz 2024-03-29T02:17:41.552 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31388.log.gz 2024-03-29T02:17:41.552 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31453.log.gz 2024-03-29T02:17:41.552 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31495.log.gz 2024-03-29T02:17:41.552 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31537.log.gz 2024-03-29T02:17:41.582 INFO:teuthology.orchestra.run.smithi007.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28549.log.gz 2024-03-29T02:17:41.583 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.35283.log.gz 2024-03-29T02:17:41.599 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30118.log.gz 2024-03-29T02:17:41.599 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30158.log.gz 2024-03-29T02:17:41.600 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30198.log.gz 2024-03-29T02:17:41.602 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30356.log.gz 2024-03-29T02:17:41.603 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30386.log.gz 2024-03-29T02:17:42.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-29T02:18:00.986 INFO:teuthology.orchestra.run.smithi007.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T02:18:01.221 INFO:teuthology.orchestra.run.smithi007.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T02:18:02.921 INFO:teuthology.orchestra.run.smithi007.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T02:18:02.922 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-29T02:18:02.923 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m21.584s 2024-03-29T02:18:02.923 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m10.218s 2024-03-29T02:18:02.923 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.540s 2024-03-29T02:18:02.923 INFO:tasks.ceph:Archiving logs... 2024-03-29T02:18:02.923 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041/remote/smithi007/log 2024-03-29T02:18:02.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T02:18:04.396 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T02:18:04.408 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T02:18:04.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T02:18:04.468 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T02:18:04.476 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T02:18:04.477 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T02:18:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-29T02:18:04.559 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T02:18:04.559 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-29T02:18:04.559 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +789us[ +789us] +/- 73ms 2024-03-29T02:18:04.559 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -16us[ -60us] +/- 79ms 2024-03-29T02:18:04.559 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -945us[ -990us] +/- 82ms 2024-03-29T02:18:04.559 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:18:04.579 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T02:18:04.589 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T02:18:04.590 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T02:18:04.614 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T02:18:04.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T02:18:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:18:04.775 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T02:18:04.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T02:18:04.812 INFO:teuthology.task.internal:Duration was 852.207749 seconds 2024-03-29T02:18:04.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T02:18:04.828 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T02:18:04.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T02:18:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T02:18:05.097 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T02:18:05.097 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-03-29T02:18:05.097 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T02:18:05.116 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T02:18:05.116 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T02:18:05.184 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T02:18:05.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T02:18:05.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T02:18:05.337 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T02:18:05.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T02:18:05.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T02:18:05.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T02:18:05.458 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-03-29T02:18:05.485 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T02:18:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:18:05.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T02:18:05.517 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T02:18:05.518 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629041/remote/smithi007 2024-03-29T02:18:05.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T02:18:05.615 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T02:18:05.615 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T02:18:05.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T02:18:05.672 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T02:18:05.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T02:18:05.700 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T02:18:05.700 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T02:18:05.714 INFO:teuthology.orchestra.run.smithi007.stdout: 265607 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 02:18 /home/ubuntu/cephtest 2024-03-29T02:18:05.715 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T02:18:05.726 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T02:18:05.765 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=ci-yingxin-crimson-load-obc-3&var-job_id_selected=7629041 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} duration: 852.2077491283417 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-03-29T02:18:05.765 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T02:18:05.891 INFO:teuthology.run:pass