2023-10-15T15:01:49.929 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-15T15:01:49.930 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_matan@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} --name matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412 -- /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412/orig.config.yaml 2023-10-15T15:01:49.977 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-15T15:01:50.060 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: null first_in_suite: false job_id: '7428412' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 984ba140e1395377d07322ca248175a686ef49e1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 984ba140e1395377d07322ca248175a686ef49e1 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 9b943423e52ce7834e53a46d2315dd8f44efe018 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4843 sha1: 984ba140e1395377d07322ca248175a686ef49e1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 9b943423e52ce7834e53a46d2315dd8f44efe018 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNjKeyBa2FxzpICfsMHddHjFSUJ1pcYtdW7Uy1MaBInIIOBsYRVwu/Kr6EW0b0LVx+F9oYFsJUxHq9lx8POqc= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-15_14:39:28 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-15T15:01:50.061 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa; will attempt to use it 2023-10-15T15:01:50.062 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa/tasks 2023-10-15T15:01:50.062 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-15T15:01:50.064 INFO:teuthology.task.internal:Checking packages... 2023-10-15T15:01:50.087 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '984ba140e1395377d07322ca248175a686ef49e1' 2023-10-15T15:01:50.088 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-15T15:01:50.088 INFO:teuthology.packaging:ref: None 2023-10-15T15:01:50.089 INFO:teuthology.packaging:tag: None 2023-10-15T15:01:50.089 INFO:teuthology.packaging:branch: main 2023-10-15T15:01:50.089 INFO:teuthology.packaging:sha1: 984ba140e1395377d07322ca248175a686ef49e1 2023-10-15T15:01:50.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2023-10-15T15:01:50.328 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6729.g984ba140 2023-10-15T15:01:50.329 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-15T15:01:50.335 INFO:teuthology.task.internal:no buildpackages task found 2023-10-15T15:01:50.336 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-15T15:01:50.351 INFO:teuthology.task.internal:Saving configuration 2023-10-15T15:01:50.367 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-15T15:01:50.376 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-15T15:01:50.402 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-15 14:57:15.944355', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNjKeyBa2FxzpICfsMHddHjFSUJ1pcYtdW7Uy1MaBInIIOBsYRVwu/Kr6EW0b0LVx+F9oYFsJUxHq9lx8POqc='} 2023-10-15T15:01:50.402 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-15T15:01:50.414 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-15T15:01:50.414 INFO:teuthology.run_tasks:Running task console_log... 2023-10-15T15:01:50.463 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc3c5c6b9d0>, signals=[15]) 2023-10-15T15:01:50.465 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-15T15:01:50.475 INFO:teuthology.task.internal:Opening connections... 2023-10-15T15:01:50.476 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:01:50.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:01:50.576 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-15T15:01:50.589 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-10-15T15:01:50.606 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-10-15T15:01:50.607 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-10-15T15:01:50.663 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2023-10-15T15:01:50.663 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2023-10-15T15:01:50.663 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2023-10-15T15:01:50.663 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2023-10-15T15:01:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2023-10-15T15:01:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2023-10-15T15:01:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-15T15:01:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-10-15T15:01:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2023-10-15T15:01:50.664 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-15T15:01:50.665 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2023-10-15T15:01:50.665 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-15T15:01:50.665 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-15T15:01:50.665 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-15T15:01:50.666 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-10-15T15:01:50.685 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-15T15:01:50.699 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-15T15:01:50.712 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-15T15:01:50.713 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-10-15T15:01:50.730 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-15T15:01:50.743 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-15T15:01:50.743 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-10-15T15:01:50.799 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-15T15:01:50.837 INFO:teuthology.run_tasks:Running task kernel... 2023-10-15T15:01:50.856 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-15T15:01:50.857 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-15T15:01:50.858 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-15T15:01:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:01:50.874 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-15T15:01:50.931 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2023-10-15T15:01:50.932 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-361.el9.x86_64 2023-10-15T15:01:50.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-10-15T15:01:53.010 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-15T15:01:54.386 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2023-10-15T15:01:56.584 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 7.9 MB 00:01 2023-10-15T15:01:58.302 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 28 MB/s | 18 MB 00:00 2023-10-15T15:02:02.128 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 13 MB/s | 5.7 MB 00:00 2023-10-15T15:02:03.703 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 14 kB 00:00 2023-10-15T15:02:07.882 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 19 MB 00:04 2023-10-15T15:02:12.514 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2023-10-15T15:02:14.669 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-15T15:02:14.669 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-15T15:02:14.768 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:02:14.778 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:02:14.779 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:02:14.779 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:02:14.779 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:02:14.779 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-375.el9 CentOS-BaseOS 5.3 M 2023-10-15T15:02:14.780 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:02:14.780 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-375.el9 CentOS-BaseOS 20 M 2023-10-15T15:02:14.780 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-375.el9 CentOS-BaseOS 37 M 2023-10-15T15:02:14.780 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-375.el9 CentOS-BaseOS 32 M 2023-10-15T15:02:14.780 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:02:14.781 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:02:14.781 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:02:14.781 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2023-10-15T15:02:14.781 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:02:14.782 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 95 M 2023-10-15T15:02:14.782 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 120 M 2023-10-15T15:02:14.782 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:02:15.263 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-375.el9.x86_64.rpm 11 MB/s | 5.3 MB 00:00 2023-10-15T15:02:15.714 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-375.el9.x86_64.rpm 21 MB/s | 20 MB 00:00 2023-10-15T15:02:16.439 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-5.14.0-375.el9.x86_64.rpm 23 MB/s | 37 MB 00:01 2023-10-15T15:02:16.981 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-core-5.14.0-375.el9.x86_6 19 MB/s | 32 MB 00:01 2023-10-15T15:02:16.983 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:02:16.984 INFO:teuthology.orchestra.run.smithi031.stdout:Total 43 MB/s | 95 MB 00:02 2023-10-15T15:02:16.984 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:02:17.423 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:02:17.423 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:02:18.477 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:02:18.477 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:02:20.977 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:02:21.646 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-375.el9.x86_64 1/4 2023-10-15T15:02:21.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-15T15:02:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-15T15:02:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-15T15:02:29.810 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-15T15:02:29.908 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:02:34.870 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:03:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:03:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:03:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:03:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-375.el9.x86_64 1/4 2023-10-15T15:03:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-15T15:03:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-15T15:03:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:03:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:03:06.403 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:03:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-375.el9.x86_64 2023-10-15T15:03:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-375.el9.x86_64 2023-10-15T15:03:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-375.el9.x86_64 2023-10-15T15:03:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-375.el9.x86_64 2023-10-15T15:03:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:03:06.404 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:03:06.896 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-10-15T15:03:07.033 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-15T15:03:07.033 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-15T15:03:07.034 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-375.el9.x86_64 2023-10-15T15:03:07.034 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-15T15:03:07.034 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-15T15:03:07.034 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-15T15:03:07.035 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2023-10-15T15:03:07.035 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-10-15T15:03:07.144 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-15T15:03:07.145 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-15T15:03:07.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-10-15T15:03:07.183 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-10-15T15:03:07.183 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-10-15T15:03:07.183 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2023-10-15T15:03:07.184 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2023-10-15T15:03:07.184 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-10-15T15:03:07.184 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-15T15:03:07.184 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2023-10-15T15:03:07.185 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2023-10-15T15:03:07.185 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-10-15T15:03:07.185 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-15T15:03:07.185 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-15T15:03:07.185 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-15T15:03:07.186 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-15T15:03:07.186 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2023-10-15T15:03:07.186 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2023-10-15T15:03:07.186 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-15T15:03:07.186 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-10-15T15:03:07.187 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-10-15T15:03:07.187 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:03:07.187 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-15T15:03:07.187 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-15T15:03:07.187 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-15T15:03:07.187 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-10-15T15:03:07.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:03:07.188 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-10-15T15:03:07.189 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-15T15:03:07.189 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-375.el9.x86_64 2023-10-15T15:03:07.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-15T15:03:07.810 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-10-15T15:03:09.146 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-15T15:03:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-10-15T15:03:09.183 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-10-15T15:03:09.199 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.mWS0x0RnNB 2023-10-15T15:03:09.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mWS0x0RnNB 2023-10-15T15:03:09.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.mWS0x0RnNB 2023-10-15T15:03:09.418 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.mWS0x0RnNB is 6KB 2023-10-15T15:03:09.466 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.mWS0x0RnNB 2023-10-15T15:03:09.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-10-15T15:03:09.550 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-15T15:03:09.550 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-15T15:03:09.551 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-15T15:03:09.551 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64.conf 2023-10-15T15:03:09.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64 2023-10-15T15:03:09.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-10-15T15:03:09.771 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-10-15T15:03:39.722 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-15T15:03:39.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-15T15:03:39.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:03:58.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-15T15:04:07.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-15T15:04:07.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:04:25.466 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-15T15:04:37.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-15T15:04:37.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:04:37.858 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-15T15:04:38.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-15T15:04:38.421 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-375.el9.x86_64"... 2023-10-15T15:04:38.421 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-15T15:04:38.436 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-375.el9.x86_64 2023-10-15T15:04:38.436 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-375.el9.x86_64 vs 5.14.0-375.el9.x86_64 2023-10-15T15:04:38.437 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-15T15:04:38.437 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-15T15:04:39.438 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-15T15:04:39.438 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-15T15:04:39.512 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-10-15T15:04:39.547 DEBUG:teuthology.parallel:result is None 2023-10-15T15:04:39.547 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-15T15:04:39.555 INFO:teuthology.task.internal:Creating test directory... 2023-10-15T15:04:39.555 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-15T15:04:39.570 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-15T15:04:39.578 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-15T15:04:39.587 INFO:teuthology.task.internal:Creating archive directory... 2023-10-15T15:04:39.587 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-15T15:04:39.629 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-15T15:04:39.637 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-15T15:04:39.638 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-15T15:04:39.707 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-15T15:04:39.724 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-15T15:04:39.725 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-15T15:04:39.733 INFO:teuthology.task.internal:Configuring sudo... 2023-10-15T15:04:39.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-15T15:04:39.765 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-15T15:04:39.774 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-15T15:04:39.774 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-15T15:04:39.818 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-15T15:04:39.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-15T15:04:39.947 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-15T15:04:40.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-15T15:04:40.072 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:04:40.072 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-15T15:04:40.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-10-15T15:04:40.242 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-15T15:04:40.557 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-15T15:04:40.565 INFO:teuthology.task.internal:Starting timer... 2023-10-15T15:04:40.565 INFO:teuthology.run_tasks:Running task pcp... 2023-10-15T15:04:40.575 INFO:teuthology.run_tasks:Running task selinux... 2023-10-15T15:04:40.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-10-15T15:04:40.642 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-10-15T15:04:40.644 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-15T15:04:40.644 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-10-15T15:04:40.660 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-10-15T15:04:40.661 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2023-10-15T15:04:40.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-15T15:04:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:04:40.730 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-15T15:04:40.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-10-15T15:04:40.799 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-15T15:04:40.809 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-15T15:04:40.809 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-15T15:04:40.826 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-15T15:04:40.828 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-15T15:10:14.949 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2023-10-15T15:10:14.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-15T15:10:14.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:10:15.049 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-15T15:10:15.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-15T15:10:15.121 INFO:teuthology.run_tasks:Running task clock... 2023-10-15T15:10:15.133 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-15T15:10:15.134 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-15T15:10:15.135 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-15T15:10:15.196 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-15T15:10:15.216 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-15T15:10:15.251 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-10-15T15:10:15.268 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-10-15T15:10:15.288 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-15T15:10:15.308 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-15T15:10:15.355 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2023-10-15T15:10:15.358 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-15T15:10:15.358 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-10-15T15:10:15.358 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:10:15.358 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:10:15.359 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:10:15.359 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:10:15.360 INFO:teuthology.run_tasks:Running task install... 2023-10-15T15:10:15.370 DEBUG:teuthology.task.install:project ceph 2023-10-15T15:10:15.371 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'}} 2023-10-15T15:10:15.371 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'} 2023-10-15T15:10:15.372 INFO:teuthology.task.install:Using flavor: crimson 2023-10-15T15:10:15.377 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-15T15:10:15.378 INFO:teuthology.task.install:extra packages: [] 2023-10-15T15:10:15.379 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '984ba140e1395377d07322ca248175a686ef49e1', 'tag': None, 'wait_for_package': False} 2023-10-15T15:10:15.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=984ba140e1395377d07322ca248175a686ef49e1 2023-10-15T15:10:15.618 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/ 2023-10-15T15:10:15.619 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6729.g984ba140 2023-10-15T15:10:15.731 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-15T15:10:15.732 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:10:15.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-15T15:10:15.765 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-15T15:10:15.765 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/984ba140e1395377d07322ca248175a686ef49e1/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-15T15:10:15.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-15T15:10:15.925 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-15T15:10:15.956 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-10-15T15:10:15.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-10-15T15:10:16.209 INFO:teuthology.orchestra.run.smithi031.stdout:72 files removed 2023-10-15T15:10:16.236 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2023-10-15T15:10:16.893 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 253 kB/s | 81 kB 00:00 2023-10-15T15:10:17.302 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 26 kB/s | 9.4 kB 00:00 2023-10-15T15:10:17.661 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2023-10-15T15:10:18.088 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-10-15T15:10:19.522 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2023-10-15T15:10:21.446 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.9 MB 00:01 2023-10-15T15:10:29.313 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 18 MB 00:06 2023-10-15T15:10:33.989 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 5.7 MB 00:01 2023-10-15T15:10:36.248 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 14 kB 00:01 2023-10-15T15:10:38.819 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 19 MB 00:02 2023-10-15T15:10:43.524 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2023-10-15T15:10:46.020 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:10:46.027 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:10:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:10:46.028 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:10:46.028 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:10:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.0.0-6729.g984ba140.el9 ceph 8.4 M 2023-10-15T15:10:46.029 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-10-15T15:10:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 3.1 M 2023-10-15T15:10:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 3.0 M 2023-10-15T15:10:46.029 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:10:46.029 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options 2023-10-15T15:10:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-15T15:10:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.0.0-6729.g984ba140.el9 ceph 6.6 M 2023-10-15T15:10:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.0.0-6729.g984ba140.el9 ceph 17 M 2023-10-15T15:10:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.0.0-6729.g984ba140.el9 ceph 24 k 2023-10-15T15:10:46.030 INFO:teuthology.orchestra.run.smithi031.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-15T15:10:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-15T15:10:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-15T15:10:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-15T15:10:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 726 k 2023-10-15T15:10:46.031 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-15T15:10:46.032 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-15T15:10:46.032 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-15T15:10:46.032 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 470 k 2023-10-15T15:10:46.032 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-15T15:10:46.032 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 4.5 M 2023-10-15T15:10:46.033 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-15T15:10:46.033 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-15T15:10:46.033 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-15T15:10:46.033 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse 2023-10-15T15:10:46.033 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:18.0.0-6729.g984ba140.el9 ceph 45 k 2023-10-15T15:10:46.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.0.0-6729.g984ba140.el9 ceph 127 k 2023-10-15T15:10:46.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.0.0-6729.g984ba140.el9 ceph 162 k 2023-10-15T15:10:46.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-15T15:10:46.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.0.0-6729.g984ba140.el9 ceph 321 k 2023-10-15T15:10:46.034 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.0.0-6729.g984ba140.el9 ceph 297 k 2023-10-15T15:10:46.035 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.0.0-6729.g984ba140.el9 ceph 99 k 2023-10-15T15:10:46.035 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-15T15:10:46.035 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-15T15:10:46.035 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:10:46.035 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:10:46.036 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:10:46.036 INFO:teuthology.orchestra.run.smithi031.stdout:Install 28 Packages 2023-10-15T15:10:46.036 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2023-10-15T15:10:46.036 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:10:46.037 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 54 M 2023-10-15T15:10:46.037 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:10:47.592 INFO:teuthology.orchestra.run.smithi031.stdout:(1/30): ceph-base-18.0.0-6729.g984ba140.el9.x86 4.9 MB/s | 6.6 MB 00:01 2023-10-15T15:10:47.650 INFO:teuthology.orchestra.run.smithi031.stdout:(2/30): ceph-selinux-18.0.0-6729.g984ba140.el9. 414 kB/s | 24 kB 00:00 2023-10-15T15:10:47.767 INFO:teuthology.orchestra.run.smithi031.stdout:(3/30): libcephfs2-18.0.0-6729.g984ba140.el9.x8 6.1 MB/s | 726 kB 00:00 2023-10-15T15:10:47.893 INFO:teuthology.orchestra.run.smithi031.stdout:(4/30): libradosstriper1-18.0.0-6729.g984ba140. 3.7 MB/s | 470 kB 00:00 2023-10-15T15:10:48.043 INFO:teuthology.orchestra.run.smithi031.stdout:(5/30): ceph-radosgw-18.0.0-6729.g984ba140.el9. 4.7 MB/s | 8.4 MB 00:01 2023-10-15T15:10:48.127 INFO:teuthology.orchestra.run.smithi031.stdout:(6/30): python3-ceph-argparse-18.0.0-6729.g984b 542 kB/s | 45 kB 00:00 2023-10-15T15:10:48.202 INFO:teuthology.orchestra.run.smithi031.stdout:(7/30): python3-ceph-common-18.0.0-6729.g984ba1 1.7 MB/s | 127 kB 00:00 2023-10-15T15:10:48.286 INFO:teuthology.orchestra.run.smithi031.stdout:(8/30): python3-cephfs-18.0.0-6729.g984ba140.el 1.9 MB/s | 162 kB 00:00 2023-10-15T15:10:48.377 INFO:teuthology.orchestra.run.smithi031.stdout:(9/30): python3-rados-18.0.0-6729.g984ba140.el9 3.4 MB/s | 321 kB 00:00 2023-10-15T15:10:48.445 INFO:teuthology.orchestra.run.smithi031.stdout:(10/30): librgw2-18.0.0-6729.g984ba140.el9.x86_ 8.1 MB/s | 4.5 MB 00:00 2023-10-15T15:10:48.470 INFO:teuthology.orchestra.run.smithi031.stdout:(11/30): python3-rbd-18.0.0-6729.g984ba140.el9. 3.2 MB/s | 297 kB 00:00 2023-10-15T15:10:48.520 INFO:teuthology.orchestra.run.smithi031.stdout:(12/30): python3-rgw-18.0.0-6729.g984ba140.el9. 1.3 MB/s | 99 kB 00:00 2023-10-15T15:10:48.578 INFO:teuthology.orchestra.run.smithi031.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2023-10-15T15:10:48.622 INFO:teuthology.orchestra.run.smithi031.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 698 kB/s | 106 kB 00:00 2023-10-15T15:10:48.796 INFO:teuthology.orchestra.run.smithi031.stdout:(15/30): ceph-common-18.0.0-6729.g984ba140.el9. 6.8 MB/s | 17 MB 00:02 2023-10-15T15:10:48.821 INFO:teuthology.orchestra.run.smithi031.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 661 kB 00:00 2023-10-15T15:10:48.846 INFO:teuthology.orchestra.run.smithi031.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 282 kB 00:00 2023-10-15T15:10:48.872 INFO:teuthology.orchestra.run.smithi031.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 555 kB/s | 41 kB 00:00 2023-10-15T15:10:49.005 INFO:teuthology.orchestra.run.smithi031.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 700 kB/s | 111 kB 00:00 2023-10-15T15:10:49.064 INFO:teuthology.orchestra.run.smithi031.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2023-10-15T15:10:49.114 INFO:teuthology.orchestra.run.smithi031.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 488 kB/s | 24 kB 00:00 2023-10-15T15:10:49.165 INFO:teuthology.orchestra.run.smithi031.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2023-10-15T15:10:49.248 INFO:teuthology.orchestra.run.smithi031.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2023-10-15T15:10:49.374 INFO:teuthology.orchestra.run.smithi031.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 6.5 MB/s | 837 kB 00:00 2023-10-15T15:10:49.415 INFO:teuthology.orchestra.run.smithi031.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2023-10-15T15:10:49.549 INFO:teuthology.orchestra.run.smithi031.stdout:(26/30): libarrow-9.0.0-9.el9.x86_64.rpm 8.1 MB/s | 4.4 MB 00:00 2023-10-15T15:10:49.641 INFO:teuthology.orchestra.run.smithi031.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2023-10-15T15:10:49.666 INFO:teuthology.orchestra.run.smithi031.stdout:(28/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 55 kB/s | 45 kB 00:00 2023-10-15T15:10:49.975 INFO:teuthology.orchestra.run.smithi031.stdout:(29/30): librados2-18.0.0-6729.g984ba140.el9.x8 7.3 MB/s | 3.1 MB 00:00 2023-10-15T15:10:50.025 INFO:teuthology.orchestra.run.smithi031.stdout:(30/30): librbd1-18.0.0-6729.g984ba140.el9.x86_ 7.9 MB/s | 3.0 MB 00:00 2023-10-15T15:10:50.029 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:10:50.029 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 54 MB 00:03 2023-10-15T15:10:50.041 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:10:50.085 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:10:50.085 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:10:50.648 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:10:50.649 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:10:52.196 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:10:52.551 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-15T15:10:52.710 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.0.0-6729.g984ba140.el9.x86_64 2/32 2023-10-15T15:10:52.787 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.0.0-6729.g984ba140.el9.x86_64 2/32 2023-10-15T15:10:52.885 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 3/32 2023-10-15T15:10:52.912 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 4/32 2023-10-15T15:10:52.975 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 4/32 2023-10-15T15:10:53.017 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-15T15:10:53.050 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-15T15:10:53.080 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-15T15:10:53.124 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.0.0-6729.g984ba140.el9. 8/32 2023-10-15T15:10:53.192 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 9/32 2023-10-15T15:10:53.216 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_6 10/32 2023-10-15T15:10:53.305 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_6 10/32 2023-10-15T15:10:53.340 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-15T15:10:53.450 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-15T15:10:53.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-15T15:10:53.524 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-15T15:10:53.555 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-15T15:10:53.583 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-15T15:10:53.618 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-15T15:10:53.651 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-15T15:10:53.979 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-15T15:10:54.023 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 20/32 2023-10-15T15:10:54.120 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 20/32 2023-10-15T15:10:54.145 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 21/32 2023-10-15T15:10:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-15T15:10:54.646 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-15T15:10:54.676 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 24/32 2023-10-15T15:10:54.748 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 24/32 2023-10-15T15:10:54.827 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 25/32 2023-10-15T15:10:54.854 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.0.0-6729.g984ba140.el9.x8 26/32 2023-10-15T15:10:56.978 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 27/32 2023-10-15T15:10:57.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 27/32 2023-10-15T15:10:57.801 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 27/32 2023-10-15T15:10:57.854 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 28/32 2023-10-15T15:10:57.926 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 28/32 2023-10-15T15:10:57.927 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-15T15:10:57.927 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-15T15:10:57.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:10:57.959 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 29/32 2023-10-15T15:11:08.101 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 29/32 2023-10-15T15:11:08.102 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-10-15T15:11:08.102 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-10-15T15:11:08.102 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-10-15T15:11:08.103 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-10-15T15:11:08.103 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-10-15T15:11:08.103 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2023-10-15T15:11:08.103 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-10-15T15:11:08.103 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:08.982 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 30/32 2023-10-15T15:11:09.008 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 30/32 2023-10-15T15:11:09.008 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:11:09.008 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-15T15:11:09.009 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-15T15:11:09.009 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-15T15:11:09.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:09.010 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-15T15:11:09.101 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-15T15:11:09.102 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-15T15:11:10.282 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-15T15:11:10.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 1/32 2023-10-15T15:11:10.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 2/32 2023-10-15T15:11:10.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 3/32 2023-10-15T15:11:10.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 4/32 2023-10-15T15:11:10.283 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 5/32 2023-10-15T15:11:10.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_6 6/32 2023-10-15T15:11:10.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 7/32 2023-10-15T15:11:10.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6729.g984ba140.el9. 8/32 2023-10-15T15:11:10.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.0.0-6729.g984ba140.el9.x8 9/32 2023-10-15T15:11:10.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 10/32 2023-10-15T15:11:10.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 11/32 2023-10-15T15:11:10.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 12/32 2023-10-15T15:11:10.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 13/32 2023-10-15T15:11:10.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-15T15:11:10.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-15T15:11:10.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-15T15:11:10.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-15T15:11:10.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-15T15:11:10.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-15T15:11:10.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-15T15:11:10.287 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-15T15:11:10.287 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-15T15:11:10.287 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-15T15:11:10.287 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-15T15:11:10.287 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-15T15:11:10.288 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-15T15:11:10.288 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-15T15:11:10.288 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-15T15:11:10.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.0.0-6729.g984ba140.el9.x86_64 29/32 2023-10-15T15:11:10.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-15T15:11:10.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 31/32 2023-10-15T15:11:11.348 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-15T15:11:11.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:11.349 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-10-15T15:11:11.349 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.349 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.350 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:11:11.350 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-15T15:11:11.350 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.350 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.351 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.351 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.351 INFO:teuthology.orchestra.run.smithi031.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-15T15:11:11.351 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-15T15:11:11.351 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-15T15:11:11.352 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-15T15:11:11.352 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.352 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-15T15:11:11.352 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-15T15:11:11.352 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-15T15:11:11.352 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.353 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-15T15:11:11.353 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.353 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-15T15:11:11.353 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-15T15:11:11.353 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-15T15:11:11.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-15T15:11:11.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.355 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.355 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:11.355 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2023-10-15T15:11:11.355 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-15T15:11:11.356 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:11.356 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:11:11.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2023-10-15T15:11:12.332 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:29 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:11:12.477 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:11:12.478 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:11:12.478 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:11:12.478 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:11:12.479 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:11:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.0.0-6729.g984ba140.el9 ceph 117 M 2023-10-15T15:11:12.479 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:11:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-15T15:11:12.479 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-15T15:11:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-15T15:11:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.0.0-6729.g984ba140.el9 ceph 161 k 2023-10-15T15:11:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-15T15:11:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-15T15:11:12.480 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-15T15:11:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-15T15:11:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-15T15:11:12.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:12.481 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:11:12.481 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:11:12.481 INFO:teuthology.orchestra.run.smithi031.stdout:Install 10 Packages 2023-10-15T15:11:12.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:12.482 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 120 M 2023-10-15T15:11:12.483 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.1 G 2023-10-15T15:11:12.483 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:11:12.988 INFO:teuthology.orchestra.run.smithi031.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.4 MB/s | 418 kB 00:00 2023-10-15T15:11:13.029 INFO:teuthology.orchestra.run.smithi031.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-10-15T15:11:13.071 INFO:teuthology.orchestra.run.smithi031.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2023-10-15T15:11:13.113 INFO:teuthology.orchestra.run.smithi031.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-10-15T15:11:13.138 INFO:teuthology.orchestra.run.smithi031.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-10-15T15:11:13.339 INFO:teuthology.orchestra.run.smithi031.stdout:(6/10): jq-1.6-15.el9.x86_64.rpm 934 kB/s | 187 kB 00:00 2023-10-15T15:11:13.414 INFO:teuthology.orchestra.run.smithi031.stdout:(7/10): libcephsqlite-18.0.0-6729.g984ba140.el9 268 kB/s | 161 kB 00:00 2023-10-15T15:11:13.531 INFO:teuthology.orchestra.run.smithi031.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2023-10-15T15:11:13.556 INFO:teuthology.orchestra.run.smithi031.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 859 kB/s | 122 kB 00:00 2023-10-15T15:11:25.547 INFO:teuthology.orchestra.run.smithi031.stdout:(10/10): ceph-test-18.0.0-6729.g984ba140.el9.x8 9.2 MB/s | 117 MB 00:12 2023-10-15T15:11:25.549 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:11:25.550 INFO:teuthology.orchestra.run.smithi031.stdout:Total 9.2 MB/s | 120 MB 00:13 2023-10-15T15:11:25.565 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:11:25.590 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:11:25.590 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:11:26.446 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:11:26.446 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:11:27.619 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:11:27.707 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-15T15:11:27.748 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-15T15:11:27.798 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-15T15:11:27.836 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-15T15:11:27.864 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-15T15:11:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-15T15:11:27.925 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-15T15:11:27.966 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-15T15:11:27.993 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 9/10 2023-10-15T15:11:45.118 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 9/10 2023-10-15T15:11:45.201 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 10/10 2023-10-15T15:11:53.013 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 10/10 2023-10-15T15:11:53.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 1/10 2023-10-15T15:11:53.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 2/10 2023-10-15T15:11:53.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-15T15:11:53.014 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-15T15:11:53.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-15T15:11:53.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-15T15:11:53.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-15T15:11:53.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-15T15:11:53.015 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-15T15:11:53.437 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-15T15:11:53.437 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:53.438 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:11:53.438 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:53.438 INFO:teuthology.orchestra.run.smithi031.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-15T15:11:53.438 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-15.el9.x86_64 2023-10-15T15:11:53.438 INFO:teuthology.orchestra.run.smithi031.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-15T15:11:53.439 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:11:53.439 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-15T15:11:53.439 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-15T15:11:53.439 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-15T15:11:53.439 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-15T15:11:53.439 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-15T15:11:53.440 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:53.440 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:11:53.743 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2023-10-15T15:11:54.343 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:11 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:11:54.500 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:11:54.503 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-10-15T15:11:54.503 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:11:54.503 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-10-15T15:11:54.503 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:11:54.503 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.0.0-6729.g984ba140.el9 ceph 6.5 k 2023-10-15T15:11:54.504 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:11:54.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.0.0-6729.g984ba140.el9 ceph 2.0 M 2023-10-15T15:11:54.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.0.0-6729.g984ba140.el9 ceph 1.5 M 2023-10-15T15:11:54.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 242 k 2023-10-15T15:11:54.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.0.0-6729.g984ba140.el9 ceph 8.8 M 2023-10-15T15:11:54.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.0.0-6729.g984ba140.el9 ceph 117 M 2023-10-15T15:11:54.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-15T15:11:54.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-15T15:11:54.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-15T15:11:54.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-15T15:11:54.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-15T15:11:54.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-15T15:11:54.507 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-15T15:11:54.507 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-15T15:11:54.507 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-15T15:11:54.507 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-15T15:11:54.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-15T15:11:54.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-15T15:11:54.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-15T15:11:54.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-15T15:11:54.508 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-15T15:11:54.509 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-15T15:11:54.509 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-15T15:11:54.509 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-15T15:11:54.509 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-15T15:11:54.509 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-15T15:11:54.510 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-15T15:11:54.510 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-15T15:11:54.510 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-15T15:11:54.510 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-15T15:11:54.510 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-15T15:11:54.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-15T15:11:54.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-15T15:11:54.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:54.511 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:11:54.511 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-10-15T15:11:54.512 INFO:teuthology.orchestra.run.smithi031.stdout:Install 33 Packages 2023-10-15T15:11:54.512 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:11:54.512 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 134 M 2023-10-15T15:11:54.513 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.4 G 2023-10-15T15:11:54.513 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:11:55.430 INFO:teuthology.orchestra.run.smithi031.stdout:(1/33): ceph-18.0.0-6729.g984ba140.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2023-10-15T15:11:55.947 INFO:teuthology.orchestra.run.smithi031.stdout:(2/33): ceph-mgr-18.0.0-6729.g984ba140.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2023-10-15T15:11:56.106 INFO:teuthology.orchestra.run.smithi031.stdout:(3/33): ceph-mds-18.0.0-6729.g984ba140.el9.x86_ 2.1 MB/s | 2.0 MB 00:00 2023-10-15T15:11:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6729.g984b 2.0 MB/s | 242 kB 00:00 2023-10-15T15:11:56.339 INFO:teuthology.orchestra.run.smithi031.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2023-10-15T15:11:56.373 INFO:teuthology.orchestra.run.smithi031.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2023-10-15T15:11:56.407 INFO:teuthology.orchestra.run.smithi031.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-15T15:11:56.449 INFO:teuthology.orchestra.run.smithi031.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-10-15T15:11:56.482 INFO:teuthology.orchestra.run.smithi031.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-15T15:11:56.525 INFO:teuthology.orchestra.run.smithi031.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-10-15T15:11:56.566 INFO:teuthology.orchestra.run.smithi031.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2023-10-15T15:11:56.608 INFO:teuthology.orchestra.run.smithi031.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-10-15T15:11:57.267 INFO:teuthology.orchestra.run.smithi031.stdout:(13/33): ceph-mon-18.0.0-6729.g984ba140.el9.x86 4.8 MB/s | 8.8 MB 00:01 2023-10-15T15:11:57.484 INFO:teuthology.orchestra.run.smithi031.stdout:(14/33): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2023-10-15T15:11:57.575 INFO:teuthology.orchestra.run.smithi031.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 166 kB/s | 51 kB 00:00 2023-10-15T15:11:57.642 INFO:teuthology.orchestra.run.smithi031.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2023-10-15T15:11:57.751 INFO:teuthology.orchestra.run.smithi031.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.3 MB/s | 358 kB 00:00 2023-10-15T15:11:57.776 INFO:teuthology.orchestra.run.smithi031.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2023-10-15T15:11:57.801 INFO:teuthology.orchestra.run.smithi031.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2023-10-15T15:11:57.827 INFO:teuthology.orchestra.run.smithi031.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2023-10-15T15:11:57.852 INFO:teuthology.orchestra.run.smithi031.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2023-10-15T15:11:57.877 INFO:teuthology.orchestra.run.smithi031.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2023-10-15T15:11:57.902 INFO:teuthology.orchestra.run.smithi031.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2023-10-15T15:11:57.953 INFO:teuthology.orchestra.run.smithi031.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2023-10-15T15:11:57.978 INFO:teuthology.orchestra.run.smithi031.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-10-15T15:11:58.020 INFO:teuthology.orchestra.run.smithi031.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2023-10-15T15:11:58.062 INFO:teuthology.orchestra.run.smithi031.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2023-10-15T15:11:58.087 INFO:teuthology.orchestra.run.smithi031.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2023-10-15T15:11:58.113 INFO:teuthology.orchestra.run.smithi031.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2023-10-15T15:11:58.171 INFO:teuthology.orchestra.run.smithi031.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-10-15T15:11:58.213 INFO:teuthology.orchestra.run.smithi031.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 481 kB/s | 20 kB 00:00 2023-10-15T15:11:58.255 INFO:teuthology.orchestra.run.smithi031.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2023-10-15T15:12:01.847 INFO:teuthology.orchestra.run.smithi031.stdout:(33/33): ceph-osd-18.0.0-6729.g984ba140.el9.x86 20 MB/s | 117 MB 00:05 2023-10-15T15:12:01.854 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:12:01.854 INFO:teuthology.orchestra.run.smithi031.stdout:Total 18 MB/s | 134 MB 00:07 2023-10-15T15:12:01.889 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:12:01.932 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:12:01.932 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:12:02.985 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:12:02.985 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:12:04.982 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:12:05.049 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-15T15:12:05.081 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-15T15:12:05.114 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-15T15:12:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-15T15:12:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-15T15:12:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-15T15:12:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-15T15:12:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-15T15:12:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-10-15T15:12:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-15T15:12:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-15T15:12:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-15T15:12:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-15T15:12:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-15T15:12:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-15T15:12:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-15T15:12:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-15T15:12:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-15T15:12:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-15T15:12:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-15T15:12:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-15T15:12:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-15T15:12:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-15T15:12:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-15T15:12:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-15T15:12:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-15T15:12:06.771 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-15T15:12:06.943 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6729.g984ba140.el9. 28/33 2023-10-15T15:12:06.975 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 29/33 2023-10-15T15:12:07.000 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 29/33 2023-10-15T15:12:07.001 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:12:07.001 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-15T15:12:07.001 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-15T15:12:07.001 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-15T15:12:07.002 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:26.340 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 30/33 2023-10-15T15:12:26.367 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 30/33 2023-10-15T15:12:26.368 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:12:26.368 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-15T15:12:26.368 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-15T15:12:26.368 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-15T15:12:26.368 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:27.299 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 31/33 2023-10-15T15:12:27.393 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 31/33 2023-10-15T15:12:27.393 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:12:27.393 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-15T15:12:27.394 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-15T15:12:27.394 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-15T15:12:27.394 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:27.533 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 32/33 2023-10-15T15:12:27.557 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 32/33 2023-10-15T15:12:27.558 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:12:27.558 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-15T15:12:27.559 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-15T15:12:27.559 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-15T15:12:27.559 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:27.624 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.0.0-6729.g984ba140.el9.x86_64 33/33 2023-10-15T15:12:37.269 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.0.0-6729.g984ba140.el9.x86_64 33/33 2023-10-15T15:12:37.269 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.0.0-6729.g984ba140.el9.x86_64 1/33 2023-10-15T15:12:37.270 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 2/33 2023-10-15T15:12:37.270 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 3/33 2023-10-15T15:12:37.270 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 4/33 2023-10-15T15:12:37.270 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 5/33 2023-10-15T15:12:37.271 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6729.g984ba140.el9. 6/33 2023-10-15T15:12:37.271 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-15T15:12:37.271 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-15T15:12:37.271 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-15T15:12:37.271 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-15T15:12:37.272 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-15T15:12:37.272 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-15T15:12:37.272 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-15T15:12:37.272 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-10-15T15:12:37.272 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-15T15:12:37.273 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-15T15:12:37.273 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-15T15:12:37.273 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-15T15:12:37.273 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-15T15:12:37.273 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-15T15:12:37.273 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-15T15:12:37.274 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-15T15:12:37.274 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-15T15:12:37.274 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-15T15:12:37.274 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-15T15:12:37.274 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-15T15:12:37.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-15T15:12:37.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-15T15:12:37.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-15T15:12:37.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-15T15:12:37.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-15T15:12:37.276 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-15T15:12:38.361 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-15T15:12:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:38.362 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:12:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:12:38.362 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:12:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:12:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:12:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:12:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:12:38.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-15T15:12:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-15T15:12:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-15T15:12:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-15T15:12:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-15T15:12:38.364 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-15T15:12:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-15T15:12:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-15T15:12:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-15T15:12:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-15T15:12:38.365 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-15T15:12:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-15T15:12:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-15T15:12:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-15T15:12:38.366 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-15T15:12:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2023-10-15T15:12:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-15T15:12:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-15T15:12:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-15T15:12:38.367 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-15T15:12:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-15T15:12:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-15T15:12:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-15T15:12:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-15T15:12:38.368 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-15T15:12:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-15T15:12:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-15T15:12:38.369 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:38.369 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:38.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2023-10-15T15:12:39.217 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:56 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:12:39.309 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:12:39.362 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:12:39.363 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:12:39.363 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:39.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2023-10-15T15:12:40.046 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:57 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:12:40.190 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:12:40.191 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:40.192 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:12:40.192 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:40.192 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:12:40.192 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 269 k 2023-10-15T15:12:40.192 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:40.193 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:12:40.193 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:40.193 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:12:40.193 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:40.194 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 269 k 2023-10-15T15:12:40.194 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 264 k 2023-10-15T15:12:40.194 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:12:40.855 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.0.0-6729.g984ba140.el9.noarch.rpm 406 kB/s | 269 kB 00:00 2023-10-15T15:12:40.856 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:12:40.856 INFO:teuthology.orchestra.run.smithi031.stdout:Total 405 kB/s | 269 kB 00:00 2023-10-15T15:12:40.857 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:12:40.862 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:12:40.862 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:12:40.865 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:12:40.866 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:12:40.941 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:12:41.455 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:12:41.545 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:12:41.931 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:12:42.274 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:12:42.274 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:42.274 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:12:42.275 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:12:42.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:42.275 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:42.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2023-10-15T15:12:43.070 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:00 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:12:43.217 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:12:43.218 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:43.218 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-15T15:12:43.218 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:43.218 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:12:43.218 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6729.g984ba140.el9 ceph 144 k 2023-10-15T15:12:43.219 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:43.219 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:12:43.219 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:43.219 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:12:43.220 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:43.220 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 144 k 2023-10-15T15:12:43.220 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 437 k 2023-10-15T15:12:43.220 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:12:43.536 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.0.0-6729.g984ba1 455 kB/s | 144 kB 00:00 2023-10-15T15:12:43.536 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:12:43.537 INFO:teuthology.orchestra.run.smithi031.stdout:Total 452 kB/s | 144 kB 00:00 2023-10-15T15:12:43.537 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:12:43.545 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:12:43.545 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:12:43.599 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:12:43.600 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:12:43.860 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:12:43.928 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6729.g984ba140. 1/1 2023-10-15T15:12:43.954 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6729.g984ba140. 1/1 2023-10-15T15:12:43.954 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:12:43.955 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-15T15:12:43.955 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6729.g984ba140. 1/1 2023-10-15T15:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:12:44.980 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:44.981 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:45.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2023-10-15T15:12:45.891 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:02 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:12:45.984 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:12:46.037 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:12:46.037 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:12:46.038 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:46.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2023-10-15T15:12:46.717 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:03 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:12:46.861 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:12:46.862 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:46.862 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:12:46.862 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:46.862 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:12:46.863 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 1.8 M 2023-10-15T15:12:46.863 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:12:46.863 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 23 k 2023-10-15T15:12:46.863 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 15 k 2023-10-15T15:12:46.864 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-15T15:12:46.864 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-15T15:12:46.864 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-15T15:12:46.865 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2023-10-15T15:12:46.865 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-15T15:12:46.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:46.866 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:12:46.866 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:12:46.867 INFO:teuthology.orchestra.run.smithi031.stdout:Install 7 Packages 2023-10-15T15:12:46.867 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:46.868 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.1 M 2023-10-15T15:12:46.868 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.6 M 2023-10-15T15:12:46.868 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:12:47.363 INFO:teuthology.orchestra.run.smithi031.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6729.g984 81 kB/s | 23 kB 00:00 2023-10-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi031.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6729.g984b 49 kB/s | 15 kB 00:00 2023-10-15T15:12:47.447 INFO:teuthology.orchestra.run.smithi031.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 108 kB/s | 9.0 kB 00:00 2023-10-15T15:12:47.539 INFO:teuthology.orchestra.run.smithi031.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 272 kB/s | 41 kB 00:00 2023-10-15T15:12:47.564 INFO:teuthology.orchestra.run.smithi031.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 263 kB/s | 31 kB 00:00 2023-10-15T15:12:47.698 INFO:teuthology.orchestra.run.smithi031.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6729.g984ba140 2.8 MB/s | 1.8 MB 00:00 2023-10-15T15:12:47.723 INFO:teuthology.orchestra.run.smithi031.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-10-15T15:12:47.725 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:12:47.725 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.4 MB/s | 2.1 MB 00:00 2023-10-15T15:12:47.726 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:12:47.735 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:12:47.735 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:12:47.811 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:12:47.811 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:12:48.118 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:12:48.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-15T15:12:48.217 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-15T15:12:48.264 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-15T15:12:48.295 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-15T15:12:48.324 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6729.g984ba140.el9.n 5/7 2023-10-15T15:12:48.571 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6729.g984ba140.el9. 6/7 2023-10-15T15:12:48.602 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarc 7/7 2023-10-15T15:12:49.166 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarc 7/7 2023-10-15T15:12:49.166 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6729.g984ba140.el9. 1/7 2023-10-15T15:12:49.167 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarc 2/7 2023-10-15T15:12:49.167 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6729.g984ba140.el9.n 3/7 2023-10-15T15:12:49.167 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-15T15:12:49.167 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-15T15:12:49.167 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-15T15:12:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-15T15:12:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:49.504 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:12:49.504 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:12:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:12:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:12:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-15T15:12:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-15T15:12:49.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-15T15:12:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-15T15:12:49.506 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:49.506 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:49.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-15T15:12:50.330 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:07 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:12:50.476 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:12:50.477 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-10-15T15:12:50.477 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:12:50.477 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-10-15T15:12:50.478 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:12:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 7.4 M 2023-10-15T15:12:50.478 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:12:50.478 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-15T15:12:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-15T15:12:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-15T15:12:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-15T15:12:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-15T15:12:50.479 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-15T15:12:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-15T15:12:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-15T15:12:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-15T15:12:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-15T15:12:50.480 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-15T15:12:50.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:50.481 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:12:50.481 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-10-15T15:12:50.482 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2023-10-15T15:12:50.482 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:50.483 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 37 M 2023-10-15T15:12:50.483 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 209 M 2023-10-15T15:12:50.483 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:12:50.613 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2023-10-15T15:12:50.646 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 549 kB/s | 17 kB 00:00 2023-10-15T15:12:50.705 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2023-10-15T15:12:50.721 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 470 kB/s | 34 kB 00:00 2023-10-15T15:12:50.789 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.0 MB/s | 205 kB 00:00 2023-10-15T15:12:50.889 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-10-15T15:12:50.981 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-10-15T15:12:51.006 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-10-15T15:12:51.365 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2023-10-15T15:12:51.398 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2023-10-15T15:12:51.423 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-10-15T15:12:51.515 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 7.1 MB/s | 7.4 MB 00:01 2023-10-15T15:12:51.518 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:12:51.518 INFO:teuthology.orchestra.run.smithi031.stdout:Total 36 MB/s | 37 MB 00:01 2023-10-15T15:12:51.519 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:12:51.542 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:12:51.542 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:12:52.006 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:12:52.007 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:12:52.737 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:12:52.831 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-15T15:12:52.870 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-15T15:12:52.968 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-15T15:12:53.006 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-15T15:12:53.190 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-15T15:12:53.222 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-15T15:12:53.310 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-15T15:12:54.004 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-15T15:12:54.130 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-15T15:12:55.456 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-15T15:12:56.528 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-15T15:12:56.554 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba 12/12 2023-10-15T15:12:59.010 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba 12/12 2023-10-15T15:12:59.010 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba 1/12 2023-10-15T15:12:59.011 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-15T15:12:59.011 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-15T15:12:59.011 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-15T15:12:59.011 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-15T15:12:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-15T15:12:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-15T15:12:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-15T15:12:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-15T15:12:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-15T15:12:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-15T15:12:59.454 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-15T15:12:59.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:59.455 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:12:59.455 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:12:59.455 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-15T15:12:59.455 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-15T15:12:59.456 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-15T15:12:59.456 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-15T15:12:59.456 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-15T15:12:59.456 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-15T15:12:59.456 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-15T15:12:59.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-15T15:12:59.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-15T15:12:59.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-15T15:12:59.457 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-15T15:12:59.457 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:12:59.457 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:12:59.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2023-10-15T15:13:00.435 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:17 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:00.582 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:00.583 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-10-15T15:13:00.583 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:13:00.583 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-10-15T15:13:00.583 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:00.584 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 47 k 2023-10-15T15:13:00.584 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:13:00.584 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-15T15:13:00.584 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-15T15:13:00.584 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-15T15:13:00.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-15T15:13:00.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-15T15:13:00.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-15T15:13:00.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-15T15:13:00.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-15T15:13:00.586 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-15T15:13:00.586 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-15T15:13:00.586 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-15T15:13:00.586 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-15T15:13:00.586 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:00.587 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:00.587 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-10-15T15:13:00.587 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-10-15T15:13:00.588 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:00.589 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2023-10-15T15:13:00.589 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2023-10-15T15:13:00.589 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:00.797 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 162 kB/s | 18 kB 00:00 2023-10-15T15:13:00.847 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2023-10-15T15:13:00.897 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-10-15T15:13:00.922 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-15T15:13:00.948 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-15T15:13:00.973 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 873 kB/s | 43 kB 00:00 2023-10-15T15:13:01.123 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2023-10-15T15:13:01.156 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 82 kB/s | 15 kB 00:00 2023-10-15T15:13:01.207 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-google-auth-2.23.2-1.el9.noarch 2.3 MB/s | 196 kB 00:00 2023-10-15T15:13:01.257 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2023-10-15T15:13:01.307 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2023-10-15T15:13:01.333 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): ceph-mgr-rook-18.0.0-6729.g984ba140.el 73 kB/s | 47 kB 00:00 2023-10-15T15:13:01.349 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 974 kB/s | 90 kB 00:00 2023-10-15T15:13:01.353 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:01.353 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2023-10-15T15:13:01.353 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:01.367 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:01.367 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:01.473 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:01.473 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:02.008 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:02.180 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-15T15:13:02.245 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-15T15:13:02.300 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-15T15:13:02.344 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-15T15:13:02.386 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-15T15:13:02.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-15T15:13:02.602 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 7/13 2023-10-15T15:13:02.671 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-15T15:13:03.439 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-15T15:13:03.492 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-15T15:13:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-15T15:13:03.572 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-15T15:13:03.595 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 13/13 2023-10-15T15:13:04.363 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 13/13 2023-10-15T15:13:04.363 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 1/13 2023-10-15T15:13:04.364 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-15T15:13:04.364 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-15T15:13:04.364 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-15T15:13:04.364 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-15T15:13:04.364 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-15T15:13:04.365 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-15T15:13:04.365 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-15T15:13:04.365 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-15T15:13:04.365 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 10/13 2023-10-15T15:13:04.365 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-15T15:13:04.366 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-15T15:13:04.763 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-15T15:13:04.763 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:04.763 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:04.764 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:13:04.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-15T15:13:04.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-15T15:13:04.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-15T15:13:04.764 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-15T15:13:04.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-15T15:13:04.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-15T15:13:04.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-15T15:13:04.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-15T15:13:04.765 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-15T15:13:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-15T15:13:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-15T15:13:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-15T15:13:04.766 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:04.766 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:05.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2023-10-15T15:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:22 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-10-15T15:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-10-15T15:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 132 k 2023-10-15T15:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-15T15:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-15T15:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-15T15:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-15T15:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-15T15:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-10-15T15:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-10-15T15:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.8 M 2023-10-15T15:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2023-10-15T15:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-10-15T15:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-10-15T15:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.4 MB/s | 548 kB 00:00 2023-10-15T15:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 533 kB/s | 58 kB 00:00 2023-10-15T15:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2023-10-15T15:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-mgr-cephadm-18.0.0-6729.g984ba140.e 216 kB/s | 132 kB 00:00 2023-10-15T15:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout:Total 10 MB/s | 6.8 MB 00:00 2023-10-15T15:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:06.852 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:07.030 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-15T15:13:07.087 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-15T15:13:07.634 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-15T15:13:07.741 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-15T15:13:07.809 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-15T15:13:07.832 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 6/6 2023-10-15T15:13:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 6/6 2023-10-15T15:13:08.682 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/6 2023-10-15T15:13:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-15T15:13:08.683 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-15T15:13:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-15T15:13:08.684 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-15T15:13:09.004 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-15T15:13:09.004 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:09.005 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:09.005 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:13:09.005 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-15T15:13:09.005 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-15T15:13:09.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-15T15:13:09.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-15T15:13:09.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-15T15:13:09.007 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:09.007 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:09.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2023-10-15T15:13:09.863 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:26 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:10.009 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:10.010 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:13:10.010 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:10.010 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:10.010 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:18.0.0-6729.g984ba140.el9 ceph 832 k 2023-10-15T15:13:10.011 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:13:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-15T15:13:10.011 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:10.011 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:10.011 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:10.012 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2023-10-15T15:13:10.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:10.012 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 910 k 2023-10-15T15:13:10.012 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 3.0 M 2023-10-15T15:13:10.012 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:10.136 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 626 kB/s | 79 kB 00:00 2023-10-15T15:13:10.795 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.0.0-6729.g984ba140.el9.x86_ 1.0 MB/s | 832 kB 00:00 2023-10-15T15:13:10.796 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:10.797 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.1 MB/s | 910 kB 00:00 2023-10-15T15:13:10.797 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:10.821 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:10.822 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:10.904 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:10.905 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:11.105 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:11.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-15T15:13:11.305 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 2/2 2023-10-15T15:13:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 2/2 2023-10-15T15:13:11.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/2 2023-10-15T15:13:12.311 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-15T15:13:12.311 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:12.312 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:12.312 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-15T15:13:12.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:12.312 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:12.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2023-10-15T15:13:13.164 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:30 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:13.311 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:13.312 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:13.312 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:13:13.312 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:13.313 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:13.313 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 262 k 2023-10-15T15:13:13.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:13.313 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:13.313 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:13.314 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:13:13.314 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:13.314 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 262 k 2023-10-15T15:13:13.314 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.2 M 2023-10-15T15:13:13.314 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:14.007 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-18.0.0-6729.g984ba140.el9.noarch.rp 377 kB/s | 262 kB 00:00 2023-10-15T15:13:14.008 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:14.009 INFO:teuthology.orchestra.run.smithi031.stdout:Total 376 kB/s | 262 kB 00:00 2023-10-15T15:13:14.009 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:14.020 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:14.020 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:14.057 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:14.058 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:14.280 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:14.351 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:13:14.367 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:13:14.367 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:13:14.368 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-15T15:13:14.368 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:15.436 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:13:15.436 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:15.436 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:15.437 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:13:15.437 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:15.437 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:15.718 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2023-10-15T15:13:16.332 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:33 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:16.478 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:16.479 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:16.480 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-15T15:13:16.480 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:16.480 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:16.480 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 127 k 2023-10-15T15:13:16.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:13:16.481 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:16.481 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 127 k 2023-10-15T15:13:16.482 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 456 k 2023-10-15T15:13:16.482 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:17.079 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.0.0-6729.g984ba140.el9.x86_64 212 kB/s | 127 kB 00:00 2023-10-15T15:13:17.079 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:17.080 INFO:teuthology.orchestra.run.smithi031.stdout:Total 211 kB/s | 127 kB 00:00 2023-10-15T15:13:17.080 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:17.086 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:17.086 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:17.147 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:17.148 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:17.301 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:17.406 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:17.845 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:18.125 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:18.125 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:18.126 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:13:18.126 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:18.126 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:18.400 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2023-10-15T15:13:19.006 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:36 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:19.102 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:19.155 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:19.156 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:19.156 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:19.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2023-10-15T15:13:19.833 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:36 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:19.978 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:19.979 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:19.979 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-15T15:13:19.980 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:19.980 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:19.980 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 31 k 2023-10-15T15:13:19.980 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:19.980 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:19.981 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:19.981 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:13:19.981 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:19.981 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2023-10-15T15:13:19.981 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 139 k 2023-10-15T15:13:19.982 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:20.249 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.0.0-6729.g984ba140.el9.x86_6 116 kB/s | 31 kB 00:00 2023-10-15T15:13:20.250 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:20.250 INFO:teuthology.orchestra.run.smithi031.stdout:Total 115 kB/s | 31 kB 00:00 2023-10-15T15:13:20.251 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:20.254 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:20.254 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:20.282 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:20.283 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:20.373 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:20.499 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:20.903 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:21.182 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:21.182 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:21.182 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:21.183 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:13:21.183 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:21.183 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:21.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2023-10-15T15:13:21.954 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:38 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:22.047 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:22.100 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:22.101 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:22.101 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:22.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2023-10-15T15:13:22.782 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:39 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:22.875 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:22.928 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:22.929 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:22.929 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:23.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2023-10-15T15:13:23.613 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:40 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:23.705 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:23.757 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:23.758 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:23.758 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:23.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2023-10-15T15:13:24.428 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:41 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:24.521 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:24.574 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:24.575 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:24.575 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:24.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2023-10-15T15:13:25.252 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:42 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:25.345 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:25.398 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:25.399 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:25.399 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:25.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2023-10-15T15:13:26.066 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:43 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:26.158 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:13:26.210 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:26.211 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-15T15:13:26.211 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:26.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2023-10-15T15:13:26.897 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:43 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:27.046 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:27.047 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-10-15T15:13:27.047 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:27.047 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:27.047 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.0.0-6729.g984ba140.el9 ceph 87 k 2023-10-15T15:13:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:27.048 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:27.048 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:27.048 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:13:27.048 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:27.049 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 87 k 2023-10-15T15:13:27.049 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 252 k 2023-10-15T15:13:27.049 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:27.261 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.0.0-6729.g984ba140.el9.x86_64.rpm 407 kB/s | 87 kB 00:00 2023-10-15T15:13:27.261 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:27.262 INFO:teuthology.orchestra.run.smithi031.stdout:Total 404 kB/s | 87 kB 00:00 2023-10-15T15:13:27.262 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:27.270 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:27.270 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:27.329 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:27.329 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:27.471 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:27.570 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:28.000 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:28.338 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:28.339 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:28.339 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:28.339 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:13:28.339 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:28.340 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:28.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-10-15T15:13:29.196 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:46 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:29.342 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:29.343 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:29.344 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-15T15:13:29.344 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:29.344 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:29.344 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.0.0-6729.g984ba140.el9 ceph 3.0 M 2023-10-15T15:13:29.344 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:29.345 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:29.345 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:29.345 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:13:29.345 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:29.345 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.0 M 2023-10-15T15:13:29.346 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 13 M 2023-10-15T15:13:29.346 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.0.0-6729.g984ba140.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2023-10-15T15:13:30.139 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.8 MB/s | 3.0 MB 00:00 2023-10-15T15:13:30.140 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:30.152 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:30.231 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:30.231 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:30.713 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:30.781 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:30.807 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:13:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-15T15:13:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-15T15:13:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-15T15:13:30.809 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:31.930 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:31.930 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:31.931 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:31.931 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:13:31.931 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:31.931 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:32.177 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2023-10-15T15:13:32.789 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:49 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:13:32.934 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:13:32.935 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:32.935 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-10-15T15:13:32.936 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:32.936 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:13:32.936 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.0.0-6729.g984ba140.el9 ceph 168 k 2023-10-15T15:13:32.936 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:32.936 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:13:32.937 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:13:32.937 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-15T15:13:32.937 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:32.937 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 168 k 2023-10-15T15:13:32.938 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 532 k 2023-10-15T15:13:32.938 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:13:33.250 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.0.0-6729.g984ba140.el9.x86_64.rpm 535 kB/s | 168 kB 00:00 2023-10-15T15:13:33.251 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:13:33.252 INFO:teuthology.orchestra.run.smithi031.stdout:Total 532 kB/s | 168 kB 00:00 2023-10-15T15:13:33.252 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:13:33.260 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:13:33.260 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:13:33.319 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:13:33.320 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:13:33.479 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:13:33.585 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:34.003 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:34.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:13:34.331 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:34.331 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:13:34.331 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:13:34.331 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:34.331 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:13:34.571 DEBUG:teuthology.parallel:result is None 2023-10-15T15:13:34.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=984ba140e1395377d07322ca248175a686ef49e1 2023-10-15T15:13:34.709 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-15T15:13:34.731 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-6729.g984ba140.el9 2023-10-15T15:13:34.732 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6729.g984ba140.el9 2023-10-15T15:13:34.732 INFO:teuthology.task.install:The correct ceph version 18.0.0-6729.g984ba140 is installed. 2023-10-15T15:13:34.732 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-15T15:13:34.733 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:34.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-15T15:13:34.804 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-15T15:13:34.804 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:34.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-10-15T15:13:34.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-15T15:13:34.946 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-15T15:13:34.946 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:34.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-15T15:13:35.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-15T15:13:35.086 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-15T15:13:35.087 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:35.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-10-15T15:13:35.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-15T15:13:35.227 INFO:teuthology.run_tasks:Running task ceph... 2023-10-15T15:13:35.287 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-15T15:13:35.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-10-15T15:13:35.318 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-15T15:13:35.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-15T15:13:35.388 INFO:tasks.ceph:Creating extra log directories... 2023-10-15T15:13:35.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-15T15:13:35.465 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-15T15:13:35.466 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-15T15:13:35.466 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': None, 'first_in_suite': False, 'job_id': '7428412', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '9b943423e52ce7834e53a46d2315dd8f44efe018'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 4843, 'sha1': '984ba140e1395377d07322ca248175a686ef49e1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '9b943423e52ce7834e53a46d2315dd8f44efe018', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNjKeyBa2FxzpICfsMHddHjFSUJ1pcYtdW7Uy1MaBInIIOBsYRVwu/Kr6EW0b0LVx+F9oYFsJUxHq9lx8POqc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '984ba140e1395377d07322ca248175a686ef49e1', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-15_14:39:28', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-15T15:13:35.467 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-15T15:13:35.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-10-15T15:13:35.583 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:35.584 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-10-15T15:13:35.638 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-15T15:13:35.639 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-10-15T15:13:35.693 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-15T15:13:35.693 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:13:35.693 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-10-15T15:13:35.694 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:13:35.694 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:13:35.694 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-15 15:13:31.052800965 +0000 2023-10-15T15:13:35.694 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-15 15:09:49.170533257 +0000 2023-10-15T15:13:35.694 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-15 15:09:49.170533257 +0000 2023-10-15T15:13:35.695 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-15 15:09:49.170533257 +0000 2023-10-15T15:13:35.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-15T15:13:35.762 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-15T15:13:35.762 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-15T15:13:35.762 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000190736 s, 2.7 MB/s 2023-10-15T15:13:35.763 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-15T15:13:35.819 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-10-15T15:13:35.874 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-15T15:13:35.875 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:13:35.875 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-10-15T15:13:35.875 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:13:35.875 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:13:35.876 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-15 15:13:31.052800965 +0000 2023-10-15T15:13:35.876 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-15 15:09:49.681525212 +0000 2023-10-15T15:13:35.876 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-15 15:09:49.681525212 +0000 2023-10-15T15:13:35.876 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-15 15:09:49.681525212 +0000 2023-10-15T15:13:35.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-15T15:13:35.942 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-15T15:13:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-15T15:13:35.943 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000190125 s, 2.7 MB/s 2023-10-15T15:13:35.944 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-15T15:13:36.000 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-10-15T15:13:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-15T15:13:36.055 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:13:36.055 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-15T15:13:36.056 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:13:36.056 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:13:36.056 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-15 15:13:31.052800965 +0000 2023-10-15T15:13:36.056 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-15 15:09:50.193517152 +0000 2023-10-15T15:13:36.056 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-15 15:09:50.193517152 +0000 2023-10-15T15:13:36.057 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-15 15:09:50.193517152 +0000 2023-10-15T15:13:36.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-15T15:13:36.124 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-15T15:13:36.124 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-15T15:13:36.124 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.00018069 s, 2.8 MB/s 2023-10-15T15:13:36.125 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-15T15:13:36.181 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-10-15T15:13:36.236 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-15T15:13:36.236 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:13:36.236 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-15T15:13:36.236 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:13:36.237 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:13:36.237 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-15 15:13:31.053800950 +0000 2023-10-15T15:13:36.237 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-15 15:09:50.707509060 +0000 2023-10-15T15:13:36.237 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-15 15:09:50.707509060 +0000 2023-10-15T15:13:36.237 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-10-15 15:09:50.707509060 +0000 2023-10-15T15:13:36.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-15T15:13:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-15T15:13:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-15T15:13:36.305 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000182536 s, 2.8 MB/s 2023-10-15T15:13:36.306 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-15T15:13:36.362 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:13:36.362 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-15T15:13:36.362 INFO:tasks.ceph:Generating config... 2023-10-15T15:13:36.363 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-15T15:13:36.364 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-15T15:13:36.364 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-10-15T15:13:36.364 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-15T15:13:36.364 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-15T15:13:36.365 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-10-15T15:13:36.365 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-15T15:13:36.365 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-15T15:13:36.366 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-15T15:13:36.366 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-10-15T15:13:36.366 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-10-15T15:13:36.366 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-10-15T15:13:36.367 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-10-15T15:13:36.367 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-15T15:13:36.367 INFO:tasks.ceph:[osd] err to stderr = True 2023-10-15T15:13:36.367 INFO:tasks.ceph:[osd] log flush on exit = True 2023-10-15T15:13:36.368 INFO:tasks.ceph:[osd] log to file = False 2023-10-15T15:13:36.368 INFO:tasks.ceph:[osd] log to stderr = True 2023-10-15T15:13:36.368 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-15T15:13:36.368 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-15T15:13:36.369 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-15T15:13:36.369 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-15T15:13:36.369 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-15T15:13:36.369 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-15T15:13:36.370 INFO:tasks.ceph:[global] auth client required = none 2023-10-15T15:13:36.370 INFO:tasks.ceph:[global] auth cluster required = none 2023-10-15T15:13:36.370 INFO:tasks.ceph:[global] auth service required = none 2023-10-15T15:13:36.370 INFO:tasks.ceph:[global] auth supported = none 2023-10-15T15:13:36.371 INFO:tasks.ceph:[global] debug asok = 0/0 2023-10-15T15:13:36.371 INFO:tasks.ceph:[global] debug auth = 0/0 2023-10-15T15:13:36.371 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-10-15T15:13:36.372 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-10-15T15:13:36.372 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-10-15T15:13:36.372 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-10-15T15:13:36.372 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-10-15T15:13:36.373 INFO:tasks.ceph:[global] debug client = 0/0 2023-10-15T15:13:36.373 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-10-15T15:13:36.373 INFO:tasks.ceph:[global] debug context = 0/0 2023-10-15T15:13:36.373 INFO:tasks.ceph:[global] debug crush = 0/0 2023-10-15T15:13:36.374 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-10-15T15:13:36.374 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-10-15T15:13:36.374 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-10-15T15:13:36.374 INFO:tasks.ceph:[global] debug filer = 0/0 2023-10-15T15:13:36.375 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-10-15T15:13:36.375 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-10-15T15:13:36.375 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-10-15T15:13:36.375 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-10-15T15:13:36.376 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-10-15T15:13:36.376 INFO:tasks.ceph:[global] debug journal = 0/0 2023-10-15T15:13:36.376 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-10-15T15:13:36.376 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-10-15T15:13:36.377 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-10-15T15:13:36.377 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-10-15T15:13:36.377 INFO:tasks.ceph:[global] debug mds = 0/0 2023-10-15T15:13:36.377 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-10-15T15:13:36.378 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-10-15T15:13:36.378 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-10-15T15:13:36.378 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-10-15T15:13:36.378 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-10-15T15:13:36.379 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-10-15T15:13:36.379 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-10-15T15:13:36.379 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-10-15T15:13:36.380 INFO:tasks.ceph:[global] debug mon = 0/0 2023-10-15T15:13:36.380 INFO:tasks.ceph:[global] debug monc = 0/0 2023-10-15T15:13:36.380 INFO:tasks.ceph:[global] debug ms = 0/0 2023-10-15T15:13:36.380 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-10-15T15:13:36.381 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-10-15T15:13:36.381 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-10-15T15:13:36.381 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-10-15T15:13:36.381 INFO:tasks.ceph:[global] debug osd = 0/0 2023-10-15T15:13:36.382 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-10-15T15:13:36.382 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-10-15T15:13:36.382 INFO:tasks.ceph:[global] debug rados = 0/0 2023-10-15T15:13:36.382 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-10-15T15:13:36.383 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-10-15T15:13:36.383 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-10-15T15:13:36.383 INFO:tasks.ceph:[global] debug refs = 0/0 2023-10-15T15:13:36.384 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-10-15T15:13:36.384 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-10-15T15:13:36.384 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-10-15T15:13:36.384 INFO:tasks.ceph:[global] debug striper = 0/0 2023-10-15T15:13:36.385 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-10-15T15:13:36.385 INFO:tasks.ceph:[global] debug timer = 0/0 2023-10-15T15:13:36.385 INFO:tasks.ceph:[global] debug tp = 0/0 2023-10-15T15:13:36.385 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-15T15:13:36.385 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-15T15:13:36.386 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-15T15:13:36.386 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-15T15:13:36.386 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-15T15:13:36.387 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-15T15:13:36.387 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-15T15:13:36.387 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-15T15:13:36.387 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-15T15:13:36.388 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-10-15T15:13:36.388 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-10-15T15:13:36.388 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-10-15T15:13:36.388 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-15T15:13:36.389 INFO:tasks.ceph:Setting up mon.a... 2023-10-15T15:13:36.389 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-15T15:13:36.484 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2023-10-15T15:13:36.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-15T15:13:36.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-15T15:13:36.571 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31')] 2023-10-15T15:13:36.572 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-15T15:13:36.573 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:36.573 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-15T15:13:36.626 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:36.709+0000 7f2e1ae39e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:36.710+0000 7f2e1ae39e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:36.710+0000 7f2e1ae39e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:36.713 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:36.713 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 139fdba4-942a-442b-a0fc-dce756691660 2023-10-15T15:13:36.713 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-10-15T15:13:36.714 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-10-15T15:13:36.714 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 139fdba4-942a-442b-a0fc-dce756691660 2023-10-15T15:13:36.714 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-10-15T15:13:36.711318+0000 2023-10-15T15:13:36.714 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-10-15T15:13:36.711318+0000 2023-10-15T15:13:36.715 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2023-10-15T15:13:36.715 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2023-10-15T15:13:36.715 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-10-15T15:13:36.715 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-15T15:13:36.716 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-15T15:13:36.731 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 139fdba4-942a-442b-a0fc-dce756691660... 2023-10-15T15:13:36.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-15T15:13:36.831 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-10-15T15:13:36.831 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-10-15T15:13:36.831 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-15T15:13:36.832 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = none 2023-10-15T15:13:36.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.832 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-10-15T15:13:36.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.832 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-10-15T15:13:36.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-10-15T15:13:36.833 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-10-15T15:13:36.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.833 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-10-15T15:13:36.834 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-10-15T15:13:36.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.834 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-15T15:13:36.834 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-10-15T15:13:36.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.835 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-10-15T15:13:36.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-15T15:13:36.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-10-15T15:13:36.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-10-15T15:13:36.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-10-15T15:13:36.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-10-15T15:13:36.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-10-15T15:13:36.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-10-15T15:13:36.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-10-15T15:13:36.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-10-15T15:13:36.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-15T15:13:36.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-15T15:13:36.838 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-10-15T15:13:36.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.838 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-15T15:13:36.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.838 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-10-15T15:13:36.842 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.842 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-10-15T15:13:36.842 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.842 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-10-15T15:13:36.842 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-10-15T15:13:36.843 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-10-15T15:13:36.843 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2023-10-15T15:13:36.843 INFO:teuthology.orchestra.run.smithi031.stdout: auth client required = none 2023-10-15T15:13:36.843 INFO:teuthology.orchestra.run.smithi031.stdout: auth cluster required = none 2023-10-15T15:13:36.843 INFO:teuthology.orchestra.run.smithi031.stdout: auth service required = none 2023-10-15T15:13:36.844 INFO:teuthology.orchestra.run.smithi031.stdout: debug asok = 0/0 2023-10-15T15:13:36.844 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 0/0 2023-10-15T15:13:36.844 INFO:teuthology.orchestra.run.smithi031.stdout: debug bdev = 0/0 2023-10-15T15:13:36.844 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 0/0 2023-10-15T15:13:36.844 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 0/0 2023-10-15T15:13:36.845 INFO:teuthology.orchestra.run.smithi031.stdout: debug buffer = 0/0 2023-10-15T15:13:36.845 INFO:teuthology.orchestra.run.smithi031.stdout: debug civetweb = 0/0 2023-10-15T15:13:36.845 INFO:teuthology.orchestra.run.smithi031.stdout: debug client = 0/0 2023-10-15T15:13:36.845 INFO:teuthology.orchestra.run.smithi031.stdout: debug compressor = 0/0 2023-10-15T15:13:36.845 INFO:teuthology.orchestra.run.smithi031.stdout: debug context = 0/0 2023-10-15T15:13:36.846 INFO:teuthology.orchestra.run.smithi031.stdout: debug crush = 0/0 2023-10-15T15:13:36.846 INFO:teuthology.orchestra.run.smithi031.stdout: debug crypto = 0/0 2023-10-15T15:13:36.846 INFO:teuthology.orchestra.run.smithi031.stdout: debug dpdk = 0/0 2023-10-15T15:13:36.846 INFO:teuthology.orchestra.run.smithi031.stdout: debug eventtrace = 0/0 2023-10-15T15:13:36.846 INFO:teuthology.orchestra.run.smithi031.stdout: debug filer = 0/0 2023-10-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: debug filestore = 0/0 2023-10-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: debug finisher = 0/0 2023-10-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: debug fuse = 0/0 2023-10-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: debug heartbeatmap = 0/0 2023-10-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: debug javaclient = 0/0 2023-10-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi031.stdout: debug journal = 0/0 2023-10-15T15:13:36.848 INFO:teuthology.orchestra.run.smithi031.stdout: debug journaler = 0/0 2023-10-15T15:13:36.848 INFO:teuthology.orchestra.run.smithi031.stdout: debug kstore = 0/0 2023-10-15T15:13:36.848 INFO:teuthology.orchestra.run.smithi031.stdout: debug leveldb = 0/0 2023-10-15T15:13:36.848 INFO:teuthology.orchestra.run.smithi031.stdout: debug lockdep = 0/0 2023-10-15T15:13:36.848 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds = 0/0 2023-10-15T15:13:36.849 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds balancer = 0/0 2023-10-15T15:13:36.849 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds locker = 0/0 2023-10-15T15:13:36.849 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds log = 0/0 2023-10-15T15:13:36.849 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds log expire = 0/0 2023-10-15T15:13:36.849 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds migrator = 0/0 2023-10-15T15:13:36.850 INFO:teuthology.orchestra.run.smithi031.stdout: debug memdb = 0/0 2023-10-15T15:13:36.850 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 0/0 2023-10-15T15:13:36.850 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgrc = 0/0 2023-10-15T15:13:36.850 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 0/0 2023-10-15T15:13:36.850 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 0/0 2023-10-15T15:13:36.851 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-10-15T15:13:36.851 INFO:teuthology.orchestra.run.smithi031.stdout: debug objclass = 0/0 2023-10-15T15:13:36.851 INFO:teuthology.orchestra.run.smithi031.stdout: debug objectcacher = 0/0 2023-10-15T15:13:36.851 INFO:teuthology.orchestra.run.smithi031.stdout: debug objecter = 0/0 2023-10-15T15:13:36.851 INFO:teuthology.orchestra.run.smithi031.stdout: debug optracker = 0/0 2023-10-15T15:13:36.852 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 0/0 2023-10-15T15:13:36.852 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 0/0 2023-10-15T15:13:36.852 INFO:teuthology.orchestra.run.smithi031.stdout: debug perfcounter = 0/0 2023-10-15T15:13:36.852 INFO:teuthology.orchestra.run.smithi031.stdout: debug rados = 0/0 2023-10-15T15:13:36.853 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 0/0 2023-10-15T15:13:36.853 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd mirror = 0/0 2023-10-15T15:13:36.853 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd replay = 0/0 2023-10-15T15:13:36.853 INFO:teuthology.orchestra.run.smithi031.stdout: debug refs = 0/0 2023-10-15T15:13:36.853 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw = 0/0 2023-10-15T15:13:36.854 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw sync = 0/0 2023-10-15T15:13:36.854 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 0/0 2023-10-15T15:13:36.854 INFO:teuthology.orchestra.run.smithi031.stdout: debug striper = 0/0 2023-10-15T15:13:36.854 INFO:teuthology.orchestra.run.smithi031.stdout: debug throttle = 0/0 2023-10-15T15:13:36.854 INFO:teuthology.orchestra.run.smithi031.stdout: debug timer = 0/0 2023-10-15T15:13:36.855 INFO:teuthology.orchestra.run.smithi031.stdout: debug tp = 0/0 2023-10-15T15:13:36.855 INFO:teuthology.orchestra.run.smithi031.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-15T15:13:36.855 INFO:teuthology.orchestra.run.smithi031.stdout: ms client mode = crc 2023-10-15T15:13:36.855 INFO:teuthology.orchestra.run.smithi031.stdout: ms cluster mode = crc 2023-10-15T15:13:36.855 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon client mode = crc 2023-10-15T15:13:36.855 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon cluster mode = crc 2023-10-15T15:13:36.856 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon service mode = crc 2023-10-15T15:13:36.856 INFO:teuthology.orchestra.run.smithi031.stdout: ms service mode = crc 2023-10-15T15:13:36.856 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 139fdba4-942a-442b-a0fc-dce756691660 2023-10-15T15:13:36.856 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.856 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-10-15T15:13:36.857 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-10-15T15:13:36.857 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.857 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-10-15T15:13:36.857 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-10-15T15:13:36.857 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-10-15T15:13:36.858 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.858 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-10-15T15:13:36.858 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-10-15T15:13:36.858 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.858 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-10-15T15:13:36.859 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-10-15T15:13:36.859 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-10-15T15:13:36.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.860 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-10-15T15:13:36.860 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-10-15T15:13:36.860 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.860 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-10-15T15:13:36.860 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.861 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-10-15T15:13:36.861 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.861 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-10-15T15:13:36.861 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.861 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-10-15T15:13:36.862 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-10-15T15:13:36.862 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.862 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-10-15T15:13:36.862 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-10-15T15:13:36.863 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 20 2023-10-15T15:13:36.863 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-10-15T15:13:36.863 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression algorithm = snappy 2023-10-15T15:13:36.863 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression mode = aggressive 2023-10-15T15:13:36.863 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-10-15T15:13:36.864 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore rocksdb cf = False 2023-10-15T15:13:36.864 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar smp = 3 2023-10-15T15:13:36.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 20 2023-10-15T15:13:36.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 20 2023-10-15T15:13:36.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug filestore = 0/0 2023-10-15T15:13:36.865 INFO:teuthology.orchestra.run.smithi031.stdout: debug journal = 0/0 2023-10-15T15:13:36.865 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-10-15T15:13:36.865 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 0/0 2023-10-15T15:13:36.865 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 10 2023-10-15T15:13:36.865 INFO:teuthology.orchestra.run.smithi031.stdout: err to stderr = True 2023-10-15T15:13:36.866 INFO:teuthology.orchestra.run.smithi031.stdout: log flush on exit = True 2023-10-15T15:13:36.866 INFO:teuthology.orchestra.run.smithi031.stdout: log to file = False 2023-10-15T15:13:36.866 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = True 2023-10-15T15:13:36.866 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-10-15T15:13:36.866 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-10-15T15:13:36.867 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-10-15T15:13:36.867 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-10-15T15:13:36.867 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-10-15T15:13:36.867 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2023-10-15T15:13:36.867 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.867 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-10-15T15:13:36.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-15T15:13:36.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-10-15T15:13:36.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-10-15T15:13:36.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-10-15T15:13:36.868 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-10-15T15:13:36.868 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-10-15T15:13:36.869 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-10-15T15:13:36.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.869 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-10-15T15:13:36.869 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-10-15T15:13:36.869 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 0/0 2023-10-15T15:13:36.870 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 0/0 2023-10-15T15:13:36.870 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-10-15T15:13:36.870 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-10-15T15:13:36.870 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-10-15T15:13:36.870 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-10-15T15:13:36.871 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-10-15T15:13:36.871 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-10-15T15:13:36.871 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-10-15T15:13:36.871 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.872 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-15T15:13:36.872 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-10-15T15:13:36.872 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-10-15T15:13:36.872 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.872 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-10-15T15:13:36.873 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-15T15:13:36.873 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-15T15:13:36.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.873 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-10-15T15:13:36.874 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-10-15T15:13:36.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.874 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-10-15T15:13:36.874 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default crimson = True 2023-10-15T15:13:36.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:36.875 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-10-15T15:13:36.875 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-10-15T15:13:36.875 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-10-15T15:13:36.875 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-10-15T15:13:36.875 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-15T15:13:36.876 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-15T15:13:36.876 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-10-15T15:13:36.877 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-15T15:13:36.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-15T15:13:36.942 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-15T15:13:36.943 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:36.943 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-15T15:13:36.958 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:36.958 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-15T15:13:37.014 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:13:37.014 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:37.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-15T15:13:37.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-15T15:13:37.100 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:37.101 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:37.155 INFO:tasks.ceph:Setting up mon nodes... 2023-10-15T15:13:37.155 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-15T15:13:37.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-15T15:13:37.262 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-15T15:13:37.264 INFO:tasks.ceph:Setting up mds nodes... 2023-10-15T15:13:37.265 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-15T15:13:37.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-15T15:13:37.318 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-15T15:13:37.336 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-15T15:13:37.337 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-15T15:13:37.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-15T15:13:37.367 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:13:37.367 INFO:tasks.ceph:role: osd.0 2023-10-15T15:13:37.368 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:13:37.368 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-15T15:13:37.445 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-15T15:13:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-15T15:13:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-15T15:13:37.445 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-15T15:13:37.445 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-15T15:13:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-15T15:13:37.446 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-15T15:13:37.446 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-15T15:13:37.446 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-15T15:13:37.447 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-15T15:13:37.941 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-15T15:13:37.944 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-15T15:13:37.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-15T15:13:38.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-15T15:13:38.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-15T15:13:38.173 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:13:38.173 INFO:tasks.ceph:role: osd.1 2023-10-15T15:13:38.174 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:13:38.174 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-15T15:13:38.251 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-15T15:13:38.251 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-15T15:13:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-15T15:13:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-15T15:13:38.252 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-15T15:13:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-15T15:13:38.252 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-15T15:13:38.253 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-15T15:13:38.253 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-15T15:13:38.253 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-15T15:13:38.708 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-15T15:13:38.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-15T15:13:38.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-15T15:13:38.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-15T15:13:38.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-15T15:13:38.917 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:13:38.917 INFO:tasks.ceph:role: osd.2 2023-10-15T15:13:38.917 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:13:38.918 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-15T15:13:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-15T15:13:38.995 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-15T15:13:38.996 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-15T15:13:38.996 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-15T15:13:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-15T15:13:38.996 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-15T15:13:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-15T15:13:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-15T15:13:38.997 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-15T15:13:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-15T15:13:39.504 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-15T15:13:39.506 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-15T15:13:39.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-15T15:13:39.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-15T15:13:39.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:40.230 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:13:40.231 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,231 seastar - Reactor backend: linux-aio 2023-10-15T15:13:40.247 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,246 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:13:40.247 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,246 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:13:40.247 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,246 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:13:40.248 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,246 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:13:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:==79915==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:13:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:==79915==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd1eb77000; bottom 0x7f7771884000; size: 0x0085ad2f3000 (574136201216) 2023-10-15T15:13:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-10-15T15:13:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:13:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:40.807 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-10-15T15:13:40.808 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-10-15T15:13:40.808 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-10-15T15:13:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:40.808 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:13:40.860 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:13:40.860 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,860 seastar - Reactor backend: linux-aio 2023-10-15T15:13:40.879 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,878 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:13:40.879 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,878 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:13:40.879 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,878 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:13:40.886 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:40,886 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:13:40.888 INFO:teuthology.orchestra.run.smithi031.stderr:==79913==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:13:40.894 INFO:teuthology.orchestra.run.smithi031.stderr:==79913==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd1eb77000; bottom 0x7f776f67d000; size: 0x0085af4fa000 (574171881472) 2023-10-15T15:13:40.895 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-10-15T15:13:40.895 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:13:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e6f12 0x261e955b 0x261eb36c 0x261edbd1 0x2601dc1f 0x1ecb029e 0x1ec25211 0x1ec2917d 0x2c9c2af5 2023-10-15T15:13:40.980 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaae5d817 0xffffffffaae59167 0xffffffffab0000aa 2023-10-15T15:13:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-10-15T15:13:44.931 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-10-15T15:13:44.931 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-10-15T15:13:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:44.932 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:13:44.932 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 139fdba4-942a-442b-a0fc-dce756691660 2023-10-15T15:13:44.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-15T15:13:44.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:45.540 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:13:45.541 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:45,540 seastar - Reactor backend: linux-aio 2023-10-15T15:13:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:45,560 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:13:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:45,560 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:13:45.561 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:45,560 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:13:45.562 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:45,560 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:13:45.563 INFO:teuthology.orchestra.run.smithi031.stderr:==80097==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:13:45.569 INFO:teuthology.orchestra.run.smithi031.stderr:==80097==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff11e2e000; bottom 0x7f01b205f000; size: 0x00fd5fdcf000 (1088235040768) 2023-10-15T15:13:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-10-15T15:13:45.570 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:13:45.595 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x22c47f90 0x22c48ba0 0x22caa89e 0x22c6f545 0x22c72b6e 0x22c8666f 0x22ca6311 0x22ca6b99 0x22c54c59 0x22c64279 0x22c65975 0x22c6634e 0x22c67439 0x22d39e28 0x22bb0cf9 0x22c782c4 0x22c79d0f 0x22c7aa78 0x22c428e4 0x22c41f3c 0x22973c88 0x249760b1 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ed6e1e8 0x1ed78355 0x1ebf5237 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:13:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-10-15T15:13:46.118 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:46.118 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-10-15T15:13:46.119 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-10-15T15:13:46.119 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-10-15T15:13:46.119 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:46.119 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:13:46.170 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:13:46.171 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:46,170 seastar - Reactor backend: linux-aio 2023-10-15T15:13:46.188 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:46,188 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:13:46.189 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:46,188 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:13:46.189 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:46,188 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:13:46.196 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:46,196 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:13:46.199 INFO:teuthology.orchestra.run.smithi031.stderr:==80095==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:13:46.205 INFO:teuthology.orchestra.run.smithi031.stderr:==80095==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff11e2e000; bottom 0x7f01afe58000; size: 0x00fd61fd6000 (1088270721024) 2023-10-15T15:13:46.205 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-10-15T15:13:46.205 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:13:46.231 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x94ce0 0x22a3a13d 0x228faaaf 0x24976b7e 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ebf6aed 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:13:46.231 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-10-15T15:13:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e6f12 0x261e955b 0x261eb36c 0x261edbd1 0x2601dc1f 0x1ecb029e 0x1ec25211 0x1ec2917d 0x2c9c2af5 2023-10-15T15:13:46.290 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaae5d817 0xffffffffaae59167 0xffffffffab0000aa 2023-10-15T15:13:50.240 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:50.240 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-10-15T15:13:50.240 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-10-15T15:13:50.240 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-10-15T15:13:50.241 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:50.241 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:13:50.241 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 139fdba4-942a-442b-a0fc-dce756691660 2023-10-15T15:13:50.291 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-15T15:13:50.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:50.867 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:13:50.868 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:50,867 seastar - Reactor backend: linux-aio 2023-10-15T15:13:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:50,887 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:13:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:50,887 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:13:50.888 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:50,887 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:13:50.889 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:50,887 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:13:50.890 INFO:teuthology.orchestra.run.smithi031.stderr:==80279==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:13:50.896 INFO:teuthology.orchestra.run.smithi031.stderr:==80279==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe988b4000; bottom 0x7f2d32ecf000; size: 0x00d1659e5000 (899353038848) 2023-10-15T15:13:50.897 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-10-15T15:13:50.897 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:13:50.922 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0xa111c 0xc384c 0x287a7 0xb738c 0x22bbf807 0x22c9fed9 0x22ca1e53 0x22ca2185 0x22ca383f 0x22ca66a6 0x22ca6b99 0x22c54c59 0x22c64279 0x22c65975 0x22c6634e 0x22c67439 0x22d39e28 0x22bb0cf9 0x22c782c4 0x22c79d0f 0x22c7aa78 0x22c428e4 0x22c41f3c 0x22973c88 0x249760b1 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ed6e1e8 0x1ed78355 0x1ebf5237 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:13:50.923 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-10-15T15:13:51.429 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:51.429 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-10-15T15:13:51.430 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-10-15T15:13:51.430 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-10-15T15:13:51.430 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:51.430 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:13:51.486 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:13:51.486 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:51,486 seastar - Reactor backend: linux-aio 2023-10-15T15:13:51.504 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:51,504 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:13:51.505 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:51,504 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:13:51.505 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:51,504 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:13:51.512 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-10-15 15:13:51,511 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:13:51.514 INFO:teuthology.orchestra.run.smithi031.stderr:==80277==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:13:51.520 INFO:teuthology.orchestra.run.smithi031.stderr:==80277==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe988b4000; bottom 0x7f2d30cc8000; size: 0x00d167bec000 (899388719104) 2023-10-15T15:13:51.521 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-10-15T15:13:51.521 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:13:51.546 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x249bc960 0x24856fb0 0x24978395 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ebf6aed 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:13:51.547 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-10-15T15:13:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e6f12 0x261e955b 0x261eb36c 0x261edbd1 0x2601dc1f 0x1ecb029e 0x1ec25211 0x1ec2917d 0x2c9c2af5 2023-10-15T15:13:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-10-15T15:13:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-10-15T15:13:55.574 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-10-15T15:13:55.574 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-10-15T15:13:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-10-15T15:13:55.575 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:13:55.575 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 139fdba4-942a-442b-a0fc-dce756691660 2023-10-15T15:13:55.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-15T15:13:55.641 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-15T15:13:55.642 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:55.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-15T15:13:55.712 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:55.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-15T15:13:55.782 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:55.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-15T15:13:55.852 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:55.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-15T15:13:55.922 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:55.923 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-15T15:13:55.978 INFO:tasks.ceph:Adding keys to all mons... 2023-10-15T15:13:55.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-15T15:13:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-10-15T15:13:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQChASxlRJ2cDxAAj2eO8dI+duocWCcvLzDMyA== 2023-10-15T15:13:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-10-15T15:13:56.047 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCkASxlFRI6OBAAiGGyvBhnBJnVk62QRGy0RQ== 2023-10-15T15:13:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-10-15T15:13:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCqASxlLqkZDxAAMXuIL4FnCpLyoUm7pBYYtA== 2023-10-15T15:13:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-10-15T15:13:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCvASxlVnPyIRAAzwASSRoQehaQUCD3m1bZwg== 2023-10-15T15:13:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-10-15T15:13:56.048 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQChASxliW/yEhAALd+8dGfVGGy1KopCGPhxDg== 2023-10-15T15:13:56.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-15T15:13:56.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-15T15:13:56.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-15T15:13:56.272 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-15T15:13:56.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-15T15:13:56.397 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-15T15:13:56.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-15T15:13:56.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-15T15:13:56.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:56.525+0000 7f3be7e1d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:56.526+0000 7f3be7e1d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-15T15:13:56.572 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:13:56.627 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-15T15:13:56.627 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-15T15:13:56.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-15T15:13:56.669 INFO:tasks.ceph.mon.a:Started 2023-10-15T15:13:56.670 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-15T15:13:56.671 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-15T15:13:56.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-15T15:13:56.674 INFO:tasks.ceph.mgr.x:Started 2023-10-15T15:13:56.675 INFO:tasks.ceph:Setting crush tunables to default 2023-10-15T15:13:56.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2023-10-15T15:13:56.758 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-10-15T15:13:56.756+0000 7f5b94644980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.758 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-10-15T15:13:56.756+0000 7f5b94644980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.758 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-10-15T15:13:56.756+0000 7f5b94644980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.759 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:56.758+0000 7f416da1bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:56.784+0000 7feb6ec17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:56.827+0000 7feb6ec17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.898 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-10-15T15:13:56.909 INFO:tasks.ceph:check_enable_crimson: True 2023-10-15T15:13:56.910 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-15T15:13:56.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-15T15:13:56.963 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:56.961+0000 7f416da1bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:56.963 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:56.961+0000 7f416da1bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:57.015+0000 7fc7d4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:57.021+0000 7fc7d4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:57.106 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-10-15T15:13:57.104+0000 7f5b8b9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-15T15:13:57.106 INFO:tasks.ceph.mon.a.smithi031.stderr:Please be aware that this feature is experimental, untested, 2023-10-15T15:13:57.107 INFO:tasks.ceph.mon.a.smithi031.stderr:unsupported, and may result in data corruption, data loss, 2023-10-15T15:13:57.107 INFO:tasks.ceph.mon.a.smithi031.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-15T15:13:57.107 INFO:tasks.ceph.mon.a.smithi031.stderr:feature with important data. 2023-10-15T15:13:57.107 INFO:tasks.ceph.mon.a.smithi031.stderr: 2023-10-15T15:13:57.449 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:57.448+0000 7f416da1bf00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-15T15:13:57.502 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:57.501+0000 7f416da1bf00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-15T15:13:57.911 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-15T15:13:57.913 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:57.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-15T15:13:57.945 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:57.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-15T15:13:58.016 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:13:58.016 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-15T15:13:58.062 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.061+0000 7f416da1bf00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-15T15:13:58.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new c83e488a-ca7c-4b06-bb74-1303f2f145eb 0 2023-10-15T15:13:58.120 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.118+0000 7f416da1bf00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-15T15:13:58.168 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.166+0000 7f416da1bf00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-15T15:13:58.226 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.225+0000 7f416da1bf00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-15T15:13:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:58.224+0000 7fa06b908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:58.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:58.231+0000 7fa06b908640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:58.273 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.271+0000 7f416da1bf00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-15T15:13:58.313 INFO:teuthology.orchestra.run.smithi031.stdout:0 2023-10-15T15:13:58.323 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 84c69051-f1f4-457c-99a8-8c47b42f8547 1 2023-10-15T15:13:58.380 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.379+0000 7f416da1bf00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-15T15:13:58.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:58.428+0000 7fcf714ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:58.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:58.435+0000 7fcf714ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:58.458 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.457+0000 7f416da1bf00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-15T15:13:58.507 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.505+0000 7f416da1bf00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-15T15:13:58.516 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-10-15T15:13:58.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 07a5101c-3da3-45b4-a60f-b9d426d2ec75 2 2023-10-15T15:13:58.564 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.563+0000 7f416da1bf00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-15T15:13:58.616 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:58.614+0000 7f416da1bf00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-15T15:13:58.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:58.631+0000 7f75a7b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:58.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:58.638+0000 7f75a7b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:58.719 INFO:teuthology.orchestra.run.smithi031.stdout:2 2023-10-15T15:13:58.730 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-15T15:13:58.804 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-15T15:13:58.805 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-15T15:13:58.847 INFO:tasks.ceph.osd.0:Started 2023-10-15T15:13:58.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-15T15:13:58.882 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-15T15:13:58.883 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-15T15:13:58.926 INFO:tasks.ceph.osd.1:Started 2023-10-15T15:13:58.926 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-15T15:13:58.965 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-15T15:13:58.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-15T15:13:59.008 INFO:tasks.ceph.osd.2:Started 2023-10-15T15:13:59.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-15T15:13:59.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.040+0000 7f416da1bf00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-15T15:13:59.214 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.212+0000 7f416da1bf00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-15T15:13:59.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:59.211+0000 7f0356715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:59.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:13:59.222+0000 7f0356715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:59.360 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.359+0000 7f416da1bf00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-15T15:13:59.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:13:59.372 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"fsid":"139fdba4-942a-442b-a0fc-dce756691660","created":"2023-10-15T15:13:56.811767+0000","modified":"2023-10-15T15:13:58.717874+0000","last_up_change":"0.000000","last_in_change":"2023-10-15T15:13:58.717874+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c83e488a-ca7c-4b06-bb74-1303f2f145eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84c69051-f1f4-457c-99a8-8c47b42f8547","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07a5101c-3da3-45b4-a60f-b9d426d2ec75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:13:59.392 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-15T15:13:59.393 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-15T15:13:59.415 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.414+0000 7f416da1bf00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-15T15:13:59.528 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.526+0000 7f416da1bf00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-15T15:13:59.631 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.630+0000 7f416da1bf00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-15T15:13:59.695 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-15T15:13:59.785 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.783+0000 7f416da1bf00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-15T15:13:59.795 INFO:teuthology.misc.health.smithi031.stderr:2023-10-15T15:13:59.791+0000 7f28c02ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:59.797 INFO:teuthology.misc.health.smithi031.stderr:2023-10-15T15:13:59.796+0000 7f28c02ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:13:59.833 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.831+0000 7f416da1bf00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-15T15:13:59.865 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-15T15:13:59.865 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":6,"fsid":"139fdba4-942a-442b-a0fc-dce756691660","created":"2023-10-15T15:13:56.811767+0000","modified":"2023-10-15T15:13:58.717874+0000","last_up_change":"0.000000","last_in_change":"2023-10-15T15:13:58.717874+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c83e488a-ca7c-4b06-bb74-1303f2f145eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84c69051-f1f4-457c-99a8-8c47b42f8547","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"07a5101c-3da3-45b4-a60f-b9d426d2ec75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:13:59.875 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-15T15:13:59.991 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:13:59.989+0000 7f416da1bf00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-15T15:14:00.849 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:14:00.848+0000 7f416da1bf00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-15T15:14:01.145 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:14:01.144+0000 7f416da1bf00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-15T15:14:01.218 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:14:01.217+0000 7f416da1bf00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-15T15:14:01.326 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:14:01.325+0000 7f416da1bf00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-15T15:14:02.346 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-15T15:14:02.344+0000 7f410dc5a640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-15T15:14:06.178 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-15T15:14:06.288 INFO:teuthology.misc.health.smithi031.stderr:2023-10-15T15:14:06.284+0000 7f629499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:06.290 INFO:teuthology.misc.health.smithi031.stderr:2023-10-15T15:14:06.289+0000 7f629499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:06.475 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-15T15:14:06.476 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":9,"fsid":"139fdba4-942a-442b-a0fc-dce756691660","created":"2023-10-15T15:13:56.811767+0000","modified":"2023-10-15T15:14:06.352646+0000","last_up_change":"2023-10-15T15:14:04.346584+0000","last_in_change":"2023-10-15T15:13:58.717874+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-15T15:14:05.397388+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c83e488a-ca7c-4b06-bb74-1303f2f145eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":28220450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":28220450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":28220450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":28220450}]},"public_addr":"172.21.15.31:6803/28220450","cluster_addr":"172.21.15.31:6802/28220450","heartbeat_back_addr":"172.21.15.31:6810/28220450","heartbeat_front_addr":"172.21.15.31:6811/28220450","state":["exists","up"]},{"osd":1,"uuid":"84c69051-f1f4-457c-99a8-8c47b42f8547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":4078229465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":4078229465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":4078229465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":4078229465}]},"public_addr":"172.21.15.31:6806/4078229465","cluster_addr":"172.21.15.31:6807/4078229465","heartbeat_back_addr":"172.21.15.31:6813/4078229465","heartbeat_front_addr":"172.21.15.31:6812/4078229465","state":["exists","up"]},{"osd":2,"uuid":"07a5101c-3da3-45b4-a60f-b9d426d2ec75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3412292895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":3412292895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3412292895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":3412292895}]},"public_addr":"172.21.15.31:6804/3412292895","cluster_addr":"172.21.15.31:6805/3412292895","heartbeat_back_addr":"172.21.15.31:6808/3412292895","heartbeat_front_addr":"172.21.15.31:6809/3412292895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:14:06.486 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-15T15:14:06.486 INFO:tasks.ceph:Creating RBD pool 2023-10-15T15:14:06.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-15T15:14:06.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:06.590+0000 7f4e4182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:06.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:06.598+0000 7f4e4182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:07.362 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2023-10-15T15:14:07.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-15T15:14:07.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:07.500+0000 7f9c4282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:07.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:07.511+0000 7f9c4282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:08.371 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-15T15:14:08.385 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-15T15:14:08.386 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-15T15:14:08.386 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-15T15:14:08.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-15T15:14:08.387 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-15T15:14:08.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:08.515+0000 7f65d499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:08.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:08.520+0000 7f65d499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:08.753 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:08.765 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3348531363},{"type":"v1","addr":"172.21.15.31:6801","nonce":3348531363}]},"active_addr":"172.21.15.31:6801/3348531363","active_change":"2023-10-15T15:14:01.338756+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2370514947}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1480278574}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":4106748975}]}]} 2023-10-15T15:14:08.767 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-15T15:14:08.768 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-15T15:14:08.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-15T15:14:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:08.876+0000 7f9608a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:08.881+0000 7f9608a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:09.068 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"fsid":"139fdba4-942a-442b-a0fc-dce756691660","created":"2023-10-15T15:13:56.811767+0000","modified":"2023-10-15T15:14:08.367996+0000","last_up_change":"2023-10-15T15:14:04.346584+0000","last_in_change":"2023-10-15T15:13:58.717874+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-15T15:14:05.397388+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-15T15:14:06.811094+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c83e488a-ca7c-4b06-bb74-1303f2f145eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":28220450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":28220450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":28220450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":28220450}]},"public_addr":"172.21.15.31:6803/28220450","cluster_addr":"172.21.15.31:6802/28220450","heartbeat_back_addr":"172.21.15.31:6810/28220450","heartbeat_front_addr":"172.21.15.31:6811/28220450","state":["exists","up"]},{"osd":1,"uuid":"84c69051-f1f4-457c-99a8-8c47b42f8547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":4078229465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":4078229465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":4078229465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":4078229465}]},"public_addr":"172.21.15.31:6806/4078229465","cluster_addr":"172.21.15.31:6807/4078229465","heartbeat_back_addr":"172.21.15.31:6813/4078229465","heartbeat_front_addr":"172.21.15.31:6812/4078229465","state":["exists","up"]},{"osd":2,"uuid":"07a5101c-3da3-45b4-a60f-b9d426d2ec75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3412292895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":3412292895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3412292895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":3412292895}]},"public_addr":"172.21.15.31:6804/3412292895","cluster_addr":"172.21.15.31:6805/3412292895","heartbeat_back_addr":"172.21.15.31:6808/3412292895","heartbeat_front_addr":"172.21.15.31:6809/3412292895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:14:09.080 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-15T15:14:09.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-15T15:14:09.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.188+0000 7f3998c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.192+0000 7f3998c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.378 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:09.378 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"fsid":"139fdba4-942a-442b-a0fc-dce756691660","created":"2023-10-15T15:13:56.811767+0000","modified":"2023-10-15T15:14:09.371252+0000","last_up_change":"2023-10-15T15:14:04.346584+0000","last_in_change":"2023-10-15T15:13:58.717874+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-15T15:14:05.397388+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-15T15:14:06.811094+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c83e488a-ca7c-4b06-bb74-1303f2f145eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":28220450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":28220450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":28220450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":28220450}]},"public_addr":"172.21.15.31:6803/28220450","cluster_addr":"172.21.15.31:6802/28220450","heartbeat_back_addr":"172.21.15.31:6810/28220450","heartbeat_front_addr":"172.21.15.31:6811/28220450","state":["exists","up"]},{"osd":1,"uuid":"84c69051-f1f4-457c-99a8-8c47b42f8547","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":4078229465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":4078229465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":4078229465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":4078229465}]},"public_addr":"172.21.15.31:6806/4078229465","cluster_addr":"172.21.15.31:6807/4078229465","heartbeat_back_addr":"172.21.15.31:6813/4078229465","heartbeat_front_addr":"172.21.15.31:6812/4078229465","state":["exists","up"]},{"osd":2,"uuid":"07a5101c-3da3-45b4-a60f-b9d426d2ec75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3412292895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":3412292895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":3412292895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":3412292895}]},"public_addr":"172.21.15.31:6804/3412292895","cluster_addr":"172.21.15.31:6805/3412292895","heartbeat_back_addr":"172.21.15.31:6808/3412292895","heartbeat_front_addr":"172.21.15.31:6809/3412292895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:14:09.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-15T15:14:09.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-15T15:14:09.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-15T15:14:09.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.507+0000 7f6f6f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.507+0000 7f827cd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.507+0000 7f901622d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.513+0000 7f827cd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.513+0000 7f901622d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.513+0000 7f6f6f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.543 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-15T15:14:09.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-15T15:14:09.544 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-15T15:14:09.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-15T15:14:09.546 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-15T15:14:09.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-15T15:14:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.697+0000 7f642671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.697+0000 7f117e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.697+0000 7fbd2006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.703+0000 7f117e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.703+0000 7f642671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:09.704+0000 7fbd2006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:09.899 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-15T15:14:09.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-15T15:14:09.924 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-15T15:14:09.929 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-15T15:14:09.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-15T15:14:09.938 DEBUG:teuthology.parallel:result is None 2023-10-15T15:14:09.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-15T15:14:09.942 DEBUG:teuthology.parallel:result is None 2023-10-15T15:14:10.912 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-15T15:14:11.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:11.033+0000 7f557862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:11.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:11.039+0000 7f557862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:11.225 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-15T15:14:11.237 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-15T15:14:12.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-15T15:14:12.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:12.358+0000 7ff48c95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:12.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:12.364+0000 7ff48c95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:12.552 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-15T15:14:12.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-15T15:14:12.564 DEBUG:teuthology.parallel:result is None 2023-10-15T15:14:12.564 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-15T15:14:12.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-15T15:14:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:12.672+0000 7fbece500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:12.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:12.676+0000 7fbece500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:12.861 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:12.862 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-15T15:14:12.873 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-15T15:14:11.343302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224755,"data_compressed":59902,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.816471"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.405599+0000","last_change":"2023-10-15T15:14:08.435973+0000","last_active":"2023-10-15T15:14:09.405599+0000","last_peered":"2023-10-15T15:14:09.405599+0000","last_clean":"2023-10-15T15:14:09.405599+0000","last_became_active":"2023-10-15T15:14:08.431709+0000","last_became_peered":"2023-10-15T15:14:08.431709+0000","last_unstale":"2023-10-15T15:14:09.405599+0000","last_undegraded":"2023-10-15T15:14:09.405599+0000","last_fullsized":"2023-10-15T15:14:09.405599+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.405020+0000","last_change":"2023-10-15T15:14:08.490229+0000","last_active":"2023-10-15T15:14:09.405020+0000","last_peered":"2023-10-15T15:14:09.405020+0000","last_clean":"2023-10-15T15:14:09.405020+0000","last_became_active":"2023-10-15T15:14:08.487898+0000","last_became_peered":"2023-10-15T15:14:08.487898+0000","last_unstale":"2023-10-15T15:14:09.405020+0000","last_undegraded":"2023-10-15T15:14:09.405020+0000","last_fullsized":"2023-10-15T15:14:09.405020+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.406230+0000","last_change":"2023-10-15T15:14:08.475043+0000","last_active":"2023-10-15T15:14:09.406230+0000","last_peered":"2023-10-15T15:14:09.406230+0000","last_clean":"2023-10-15T15:14:09.406230+0000","last_became_active":"2023-10-15T15:14:08.472555+0000","last_became_peered":"2023-10-15T15:14:08.472555+0000","last_unstale":"2023-10-15T15:14:09.406230+0000","last_undegraded":"2023-10-15T15:14:09.406230+0000","last_fullsized":"2023-10-15T15:14:09.406230+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.404598+0000","last_change":"2023-10-15T15:14:08.437801+0000","last_active":"2023-10-15T15:14:09.404598+0000","last_peered":"2023-10-15T15:14:09.404598+0000","last_clean":"2023-10-15T15:14:09.404598+0000","last_became_active":"2023-10-15T15:14:08.432637+0000","last_became_peered":"2023-10-15T15:14:08.432637+0000","last_unstale":"2023-10-15T15:14:09.404598+0000","last_undegraded":"2023-10-15T15:14:09.404598+0000","last_fullsized":"2023-10-15T15:14:09.404598+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-15T15:14:08.465268+0000","last_change":"2023-10-15T15:14:08.465350+0000","last_active":"2023-10-15T15:14:08.465268+0000","last_peered":"2023-10-15T15:14:08.465268+0000","last_clean":"2023-10-15T15:14:07.360530+0000","last_became_active":"2023-10-15T15:14:08.462231+0000","last_became_peered":"2023-10-15T15:14:08.462231+0000","last_unstale":"2023-10-15T15:14:08.465268+0000","last_undegraded":"2023-10-15T15:14:08.465268+0000","last_fullsized":"2023-10-15T15:14:08.465268+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.380552+0000","last_change":"2023-10-15T15:14:08.481488+0000","last_active":"2023-10-15T15:14:09.380552+0000","last_peered":"2023-10-15T15:14:09.380552+0000","last_clean":"2023-10-15T15:14:09.380552+0000","last_became_active":"2023-10-15T15:14:08.479049+0000","last_became_peered":"2023-10-15T15:14:08.479049+0000","last_unstale":"2023-10-15T15:14:09.380552+0000","last_undegraded":"2023-10-15T15:14:09.380552+0000","last_fullsized":"2023-10-15T15:14:09.380552+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.379384+0000","last_change":"2023-10-15T15:14:08.452736+0000","last_active":"2023-10-15T15:14:09.379384+0000","last_peered":"2023-10-15T15:14:09.379384+0000","last_clean":"2023-10-15T15:14:09.379384+0000","last_became_active":"2023-10-15T15:14:08.450694+0000","last_became_peered":"2023-10-15T15:14:08.450694+0000","last_unstale":"2023-10-15T15:14:09.379384+0000","last_undegraded":"2023-10-15T15:14:09.379384+0000","last_fullsized":"2023-10-15T15:14:09.379384+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.405730+0000","last_change":"2023-10-15T15:14:08.467154+0000","last_active":"2023-10-15T15:14:09.405730+0000","last_peered":"2023-10-15T15:14:09.405730+0000","last_clean":"2023-10-15T15:14:09.405730+0000","last_became_active":"2023-10-15T15:14:08.463215+0000","last_became_peered":"2023-10-15T15:14:08.463215+0000","last_unstale":"2023-10-15T15:14:09.405730+0000","last_undegraded":"2023-10-15T15:14:09.405730+0000","last_fullsized":"2023-10-15T15:14:09.405730+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.401289+0000","last_change":"2023-10-15T15:14:07.430006+0000","last_active":"2023-10-15T15:14:09.401289+0000","last_peered":"2023-10-15T15:14:09.401289+0000","last_clean":"2023-10-15T15:14:09.401289+0000","last_became_active":"2023-10-15T15:14:07.423612+0000","last_became_peered":"2023-10-15T15:14:07.423612+0000","last_unstale":"2023-10-15T15:14:09.401289+0000","last_undegraded":"2023-10-15T15:14:09.401289+0000","last_fullsized":"2023-10-15T15:14:09.401289+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:06.352646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:06.352646+0000","last_clean_scrub_stamp":"2023-10-15T15:14:06.352646+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29951,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29951,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-15T15:14:12.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-15T15:14:12.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:12.981+0000 7fce9cc13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:12.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:12.986+0000 7fce9cc13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:13.168 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:13.168 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-15T15:14:13.180 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-15T15:14:11.343302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224755,"data_compressed":59902,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.816471"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.405599+0000","last_change":"2023-10-15T15:14:08.435973+0000","last_active":"2023-10-15T15:14:09.405599+0000","last_peered":"2023-10-15T15:14:09.405599+0000","last_clean":"2023-10-15T15:14:09.405599+0000","last_became_active":"2023-10-15T15:14:08.431709+0000","last_became_peered":"2023-10-15T15:14:08.431709+0000","last_unstale":"2023-10-15T15:14:09.405599+0000","last_undegraded":"2023-10-15T15:14:09.405599+0000","last_fullsized":"2023-10-15T15:14:09.405599+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.405020+0000","last_change":"2023-10-15T15:14:08.490229+0000","last_active":"2023-10-15T15:14:09.405020+0000","last_peered":"2023-10-15T15:14:09.405020+0000","last_clean":"2023-10-15T15:14:09.405020+0000","last_became_active":"2023-10-15T15:14:08.487898+0000","last_became_peered":"2023-10-15T15:14:08.487898+0000","last_unstale":"2023-10-15T15:14:09.405020+0000","last_undegraded":"2023-10-15T15:14:09.405020+0000","last_fullsized":"2023-10-15T15:14:09.405020+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.406230+0000","last_change":"2023-10-15T15:14:08.475043+0000","last_active":"2023-10-15T15:14:09.406230+0000","last_peered":"2023-10-15T15:14:09.406230+0000","last_clean":"2023-10-15T15:14:09.406230+0000","last_became_active":"2023-10-15T15:14:08.472555+0000","last_became_peered":"2023-10-15T15:14:08.472555+0000","last_unstale":"2023-10-15T15:14:09.406230+0000","last_undegraded":"2023-10-15T15:14:09.406230+0000","last_fullsized":"2023-10-15T15:14:09.406230+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.404598+0000","last_change":"2023-10-15T15:14:08.437801+0000","last_active":"2023-10-15T15:14:09.404598+0000","last_peered":"2023-10-15T15:14:09.404598+0000","last_clean":"2023-10-15T15:14:09.404598+0000","last_became_active":"2023-10-15T15:14:08.432637+0000","last_became_peered":"2023-10-15T15:14:08.432637+0000","last_unstale":"2023-10-15T15:14:09.404598+0000","last_undegraded":"2023-10-15T15:14:09.404598+0000","last_fullsized":"2023-10-15T15:14:09.404598+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-10-15T15:14:08.465268+0000","last_change":"2023-10-15T15:14:08.465350+0000","last_active":"2023-10-15T15:14:08.465268+0000","last_peered":"2023-10-15T15:14:08.465268+0000","last_clean":"2023-10-15T15:14:07.360530+0000","last_became_active":"2023-10-15T15:14:08.462231+0000","last_became_peered":"2023-10-15T15:14:08.462231+0000","last_unstale":"2023-10-15T15:14:08.465268+0000","last_undegraded":"2023-10-15T15:14:08.465268+0000","last_fullsized":"2023-10-15T15:14:08.465268+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.380552+0000","last_change":"2023-10-15T15:14:08.481488+0000","last_active":"2023-10-15T15:14:09.380552+0000","last_peered":"2023-10-15T15:14:09.380552+0000","last_clean":"2023-10-15T15:14:09.380552+0000","last_became_active":"2023-10-15T15:14:08.479049+0000","last_became_peered":"2023-10-15T15:14:08.479049+0000","last_unstale":"2023-10-15T15:14:09.380552+0000","last_undegraded":"2023-10-15T15:14:09.380552+0000","last_fullsized":"2023-10-15T15:14:09.380552+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.379384+0000","last_change":"2023-10-15T15:14:08.452736+0000","last_active":"2023-10-15T15:14:09.379384+0000","last_peered":"2023-10-15T15:14:09.379384+0000","last_clean":"2023-10-15T15:14:09.379384+0000","last_became_active":"2023-10-15T15:14:08.450694+0000","last_became_peered":"2023-10-15T15:14:08.450694+0000","last_unstale":"2023-10-15T15:14:09.379384+0000","last_undegraded":"2023-10-15T15:14:09.379384+0000","last_fullsized":"2023-10-15T15:14:09.379384+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.405730+0000","last_change":"2023-10-15T15:14:08.467154+0000","last_active":"2023-10-15T15:14:09.405730+0000","last_peered":"2023-10-15T15:14:09.405730+0000","last_clean":"2023-10-15T15:14:09.405730+0000","last_became_active":"2023-10-15T15:14:08.463215+0000","last_became_peered":"2023-10-15T15:14:08.463215+0000","last_unstale":"2023-10-15T15:14:09.405730+0000","last_undegraded":"2023-10-15T15:14:09.405730+0000","last_fullsized":"2023-10-15T15:14:09.405730+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:07.360530+0000","last_clean_scrub_stamp":"2023-10-15T15:14:07.360530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:14:09.401289+0000","last_change":"2023-10-15T15:14:07.430006+0000","last_active":"2023-10-15T15:14:09.401289+0000","last_peered":"2023-10-15T15:14:09.401289+0000","last_clean":"2023-10-15T15:14:09.401289+0000","last_became_active":"2023-10-15T15:14:07.423612+0000","last_became_peered":"2023-10-15T15:14:07.423612+0000","last_unstale":"2023-10-15T15:14:09.401289+0000","last_undegraded":"2023-10-15T15:14:09.401289+0000","last_fullsized":"2023-10-15T15:14:09.401289+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:14:06.352646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:14:06.352646+0000","last_clean_scrub_stamp":"2023-10-15T15:14:06.352646+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29951,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29951,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-15T15:14:13.181 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-15T15:14:13.181 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-15T15:14:13.181 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-15T15:14:13.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:13.288+0000 7f8d5366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:13.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:14:13.292+0000 7f8d5366d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:14:13.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:13.514 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-15T15:14:13.525 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-15T15:14:13.526 INFO:teuthology.run_tasks:Running task cbt... 2023-10-15T15:14:13.539 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi031.front.sepia.ceph.com', 'osds': ['ubuntu@smithi031.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi031.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi031.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi031.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-10-15T15:14:13.539 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-10-15T15:14:13.557 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:14:13.558 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-15T15:14:13.610 DEBUG:teuthology.orchestra.run.smithi031:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-10-15T15:14:13.665 INFO:teuthology.orchestra.run.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-10-15T15:14:14.031 INFO:tasks.cbt:Installing collectl 2023-10-15T15:14:14.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-10-15T15:14:14.136 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-10-15 15:14:14-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-10-15T15:14:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2023-10-15T15:14:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-10-15T15:14:14.515 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-15T15:14:14.515 INFO:teuthology.orchestra.run.smithi031.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlLAHGqbP9qzjK2_O5QTWlQqqFXhh944SeaYoS7U2kpfDCrmH5Xc2FMB6CCowEK02NKq8wxs2AedDZhPmoVDfAUJlPFA%3D%3D&use_mirror=cfhcable&r= [following] 2023-10-15T15:14:14.515 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-10-15 15:14:14-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlLAHGqbP9qzjK2_O5QTWlQqqFXhh944SeaYoS7U2kpfDCrmH5Xc2FMB6CCowEK02NKq8wxs2AedDZhPmoVDfAUJlPFA%3D%3D&use_mirror=cfhcable&r= 2023-10-15T15:14:14.525 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-10-15T15:14:14.596 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-10-15T15:14:14.832 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-15T15:14:14.833 INFO:teuthology.orchestra.run.smithi031.stderr:Location: https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-10-15T15:14:14.833 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-10-15 15:14:14-- https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-10-15T15:14:14.843 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)... 146.71.73.6 2023-10-15T15:14:14.877 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)|146.71.73.6|:443... connected. 2023-10-15T15:14:15.048 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-15T15:14:15.049 INFO:teuthology.orchestra.run.smithi031.stderr:Length: 620693 (606K) [application/x-gzip] 2023-10-15T15:14:15.049 INFO:teuthology.orchestra.run.smithi031.stderr:Saving to: ‘collectl.tar.gz’ 2023-10-15T15:14:15.049 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:14:15.082 INFO:teuthology.orchestra.run.smithi031.stderr: 0K .......... .......... .......... .......... .......... 8% 1.45M 0s 2023-10-15T15:14:15.116 INFO:teuthology.orchestra.run.smithi031.stderr: 50K .......... .......... .......... .......... .......... 16% 1.49M 0s 2023-10-15T15:14:15.116 INFO:teuthology.orchestra.run.smithi031.stderr: 100K .......... .......... .......... .......... .......... 24% 55.4M 0s 2023-10-15T15:14:15.151 INFO:teuthology.orchestra.run.smithi031.stderr: 150K .......... .......... .......... .......... .......... 32% 1.50M 0s 2023-10-15T15:14:15.152 INFO:teuthology.orchestra.run.smithi031.stderr: 200K .......... .......... .......... .......... .......... 41% 130M 0s 2023-10-15T15:14:15.152 INFO:teuthology.orchestra.run.smithi031.stderr: 250K .......... .......... .......... .......... .......... 49% 112M 0s 2023-10-15T15:14:15.152 INFO:teuthology.orchestra.run.smithi031.stderr: 300K .......... .......... .......... .......... .......... 57% 85.4M 0s 2023-10-15T15:14:15.184 INFO:teuthology.orchestra.run.smithi031.stderr: 350K .......... .......... .......... .......... .......... 65% 1.54M 0s 2023-10-15T15:14:15.185 INFO:teuthology.orchestra.run.smithi031.stderr: 400K .......... .......... .......... .......... .......... 74% 101M 0s 2023-10-15T15:14:15.185 INFO:teuthology.orchestra.run.smithi031.stderr: 450K .......... .......... .......... .......... .......... 82% 110M 0s 2023-10-15T15:14:15.185 INFO:teuthology.orchestra.run.smithi031.stderr: 500K .......... .......... .......... .......... .......... 90% 113M 0s 2023-10-15T15:14:15.355 INFO:teuthology.orchestra.run.smithi031.stderr: 550K .......... .......... .......... .......... .......... 98% 291K 0s 2023-10-15T15:14:15.355 INFO:teuthology.orchestra.run.smithi031.stderr: 600K ...... 100% 11.4T=0.3s 2023-10-15T15:14:15.355 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:14:15.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15 15:14:15 (1.93 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-10-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:14:15.421 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/ 2023-10-15T15:14:15.421 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/collectl.conf 2023-10-15T15:14:15.421 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/ 2023-10-15T15:14:15.422 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/SlowProc.html 2023-10-15T15:14:15.422 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-10-15T15:14:15.422 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/WhySummary.html 2023-10-15T15:14:15.423 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Socket.html 2023-10-15T15:14:15.423 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/colmux.html 2023-10-15T15:14:15.423 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Tutorial.html 2023-10-15T15:14:15.423 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Gexpr.html 2023-10-15T15:14:15.423 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Import.html 2023-10-15T15:14:15.424 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/style.css 2023-10-15T15:14:15.424 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-10-15T15:14:15.424 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-10-15T15:14:15.424 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/FileNaming.html 2023-10-15T15:14:15.424 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/TheMath.html 2023-10-15T15:14:15.425 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Environmental.html 2023-10-15T15:14:15.425 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Process.html 2023-10-15T15:14:15.425 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-10-15T15:14:15.425 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/HiResTime.html 2023-10-15T15:14:15.425 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-10-15T15:14:15.426 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/slub.jpg 2023-10-15T15:14:15.426 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Lustre.html 2023-10-15T15:14:15.426 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Documentation.html 2023-10-15T15:14:15.426 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Inodes.html 2023-10-15T15:14:15.426 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Graphite.html 2023-10-15T15:14:15.427 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/InputFiles.html 2023-10-15T15:14:15.427 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Export.html 2023-10-15T15:14:15.427 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Interrupts.html 2023-10-15T15:14:15.427 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Startup.html 2023-10-15T15:14:15.427 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Performance.html 2023-10-15T15:14:15.428 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Messages.html 2023-10-15T15:14:15.428 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Logging.html 2023-10-15T15:14:15.428 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data-detail.html 2023-10-15T15:14:15.428 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/CPUs.html 2023-10-15T15:14:15.428 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Matrix.html 2023-10-15T15:14:15.429 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data-brief.html 2023-10-15T15:14:15.429 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-10-15T15:14:15.429 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Misc.html 2023-10-15T15:14:15.429 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Examples.html 2023-10-15T15:14:15.429 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OpenStack.html 2023-10-15T15:14:15.430 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-10-15T15:14:15.430 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-10-15T15:14:15.430 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-10-15T15:14:15.430 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-10-15T15:14:15.430 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data.html 2023-10-15T15:14:15.431 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Exceptions.html 2023-10-15T15:14:15.431 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-10-15T15:14:15.431 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-10-15T15:14:15.432 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-10-15T15:14:15.432 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-10-15T15:14:15.432 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Playback.html 2023-10-15T15:14:15.433 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Hello.html 2023-10-15T15:14:15.433 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Sockets.html 2023-10-15T15:14:15.433 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Network.html 2023-10-15T15:14:15.434 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Infiniband.html 2023-10-15T15:14:15.434 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Memory.html 2023-10-15T15:14:15.435 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-10-15T15:14:15.435 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Disks.html 2023-10-15T15:14:15.435 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-10-15T15:14:15.436 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/service/ 2023-10-15T15:14:15.436 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/service/collectl.service 2023-10-15T15:14:15.436 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/ 2023-10-15T15:14:15.437 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl-suse 2023-10-15T15:14:15.437 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl-debian 2023-10-15T15:14:15.437 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl-generic 2023-10-15T15:14:15.438 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl 2023-10-15T15:14:15.438 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/misc.ph 2023-10-15T15:14:15.439 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/gexpr.ph 2023-10-15T15:14:15.439 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/statsd.ph 2023-10-15T15:14:15.439 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/envrules.std 2023-10-15T15:14:15.439 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/hello.ph 2023-10-15T15:14:15.440 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/vmsum.ph 2023-10-15T15:14:15.440 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/proctree.ph 2023-10-15T15:14:15.440 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/README-WINDOWS 2023-10-15T15:14:15.440 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/formatit.ph 2023-10-15T15:14:15.442 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/colmux 2023-10-15T15:14:15.442 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/INSTALL 2023-10-15T15:14:15.443 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/graphite.ph 2023-10-15T15:14:15.443 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/RELEASE-collectl 2023-10-15T15:14:15.443 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/vmstat.ph 2023-10-15T15:14:15.443 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/README 2023-10-15T15:14:15.443 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/man1/ 2023-10-15T15:14:15.444 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/man1/collectl.1 2023-10-15T15:14:15.444 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/man1/colmux.1 2023-10-15T15:14:15.444 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/lexpr.ph 2023-10-15T15:14:15.444 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/client.pl 2023-10-15T15:14:15.444 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/collectl 2023-10-15T15:14:15.444 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/COPYING 2023-10-15T15:14:15.445 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/vnet.ph 2023-10-15T15:14:15.445 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/GPL 2023-10-15T15:14:15.445 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/UNINSTALL 2023-10-15T15:14:15.445 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/ARTISTIC 2023-10-15T15:14:15.486 INFO:teuthology.orchestra.run.smithi031.stderr:./INSTALL: line 90: chkconfig: command not found 2023-10-15T15:14:15.487 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-10-15T15:14:16.098 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:33 ago on Sun 15 Oct 2023 03:10:43 PM UTC. 2023-10-15T15:14:16.193 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-10-15T15:14:16.195 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-10-15T15:14:16.252 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-15T15:14:16.253 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:14:16.253 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-15T15:14:16.253 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:14:16.254 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-15T15:14:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: librbd-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 23 k 2023-10-15T15:14:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-10-15T15:14:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-10-15T15:14:16.254 INFO:teuthology.orchestra.run.smithi031.stdout: perf x86_64 5.14.0-375.el9 CentOS-AppStream 7.3 M 2023-10-15T15:14:16.255 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-15T15:14:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: libradospp-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 30 k 2023-10-15T15:14:16.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:16.255 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-15T15:14:16.255 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-15T15:14:16.256 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2023-10-15T15:14:16.256 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:16.256 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 7.5 M 2023-10-15T15:14:16.257 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 11 M 2023-10-15T15:14:16.257 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-15T15:14:16.868 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): librbd-devel-18.0.0-6729.g984ba140.el9.x 108 kB/s | 23 kB 00:00 2023-10-15T15:14:16.977 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): perf-5.14.0-375.el9.x86_64.rpm 23 MB/s | 7.3 MB 00:00 2023-10-15T15:14:17.002 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): libradospp-devel-18.0.0-6729.g984ba140.e 87 kB/s | 30 kB 00:00 2023-10-15T15:14:17.102 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 105 kB/s | 13 kB 00:00 2023-10-15T15:14:17.219 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 340 kB/s | 118 kB 00:00 2023-10-15T15:14:17.221 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:14:17.221 INFO:teuthology.orchestra.run.smithi031.stdout:Total 7.7 MB/s | 7.5 MB 00:00 2023-10-15T15:14:17.222 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-15T15:14:17.244 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-15T15:14:17.245 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-15T15:14:17.406 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-15T15:14:17.406 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-15T15:14:17.717 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-15T15:14:17.805 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-10-15T15:14:17.843 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-10-15T15:14:17.874 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradospp-devel-2:18.0.0-6729.g984ba140.el9.x86_64 3/5 2023-10-15T15:14:18.229 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librbd-devel-2:18.0.0-6729.g984ba140.el9.x86_64 4/5 2023-10-15T15:14:18.356 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : perf-5.14.0-375.el9.x86_64 5/5 2023-10-15T15:14:19.265 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: perf-5.14.0-375.el9.x86_64 5/5 2023-10-15T15:14:19.266 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradospp-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/5 2023-10-15T15:14:19.266 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2/5 2023-10-15T15:14:19.266 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : perf-5.14.0-375.el9.x86_64 3/5 2023-10-15T15:14:19.266 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-10-15T15:14:19.616 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-10-15T15:14:19.616 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:19.617 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-15T15:14:19.617 INFO:teuthology.orchestra.run.smithi031.stdout: libradospp-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:14:19.617 INFO:teuthology.orchestra.run.smithi031.stdout: librbd-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:14:19.618 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh-2.34-7.el9.x86_64 2023-10-15T15:14:19.618 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-10-15T15:14:19.619 INFO:teuthology.orchestra.run.smithi031.stdout: perf-5.14.0-375.el9.x86_64 2023-10-15T15:14:19.619 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-15T15:14:19.619 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-15T15:14:20.001 INFO:tasks.cbt:benchmark: radosbench 2023-10-15T15:14:20.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-10-15T15:14:20.034 DEBUG:teuthology.orchestra.run.smithi031:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-15T15:14:20.465 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - Settings.cluster: 2023-10-15T15:14:20.466 INFO:teuthology.orchestra.run.smithi031.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-10-15T15:14:20.466 INFO:teuthology.orchestra.run.smithi031.stderr: 'clients': ['ubuntu@smithi031.front.sepia.ceph.com'], 2023-10-15T15:14:20.466 INFO:teuthology.orchestra.run.smithi031.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-10-15T15:14:20.466 INFO:teuthology.orchestra.run.smithi031.stderr: 'head': 'ubuntu@smithi031.front.sepia.ceph.com', 2023-10-15T15:14:20.467 INFO:teuthology.orchestra.run.smithi031.stderr: 'is_teuthology': True, 2023-10-15T15:14:20.467 INFO:teuthology.orchestra.run.smithi031.stderr: 'iterations': 1, 2023-10-15T15:14:20.467 INFO:teuthology.orchestra.run.smithi031.stderr: 'mons': ['ubuntu@smithi031.front.sepia.ceph.com'], 2023-10-15T15:14:20.467 INFO:teuthology.orchestra.run.smithi031.stderr: 'osds': ['ubuntu@smithi031.front.sepia.ceph.com'], 2023-10-15T15:14:20.468 INFO:teuthology.orchestra.run.smithi031.stderr: 'osds_per_node': 3, 2023-10-15T15:14:20.468 INFO:teuthology.orchestra.run.smithi031.stderr: 'pid_dir': '/var/run/ceph', 2023-10-15T15:14:20.468 INFO:teuthology.orchestra.run.smithi031.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-10-15T15:14:20.468 INFO:teuthology.orchestra.run.smithi031.stderr: 'pgp_size': 256, 2023-10-15T15:14:20.468 INFO:teuthology.orchestra.run.smithi031.stderr: 'replication': 'replicated'}}, 2023-10-15T15:14:20.469 INFO:teuthology.orchestra.run.smithi031.stderr: 'rebuild_every_test': False, 2023-10-15T15:14:20.469 INFO:teuthology.orchestra.run.smithi031.stderr: 'rgws': {'ubuntu@smithi031.front.sepia.ceph.com': None}, 2023-10-15T15:14:20.469 INFO:teuthology.orchestra.run.smithi031.stderr: 'tmp_dir': '/tmp/cbt', 2023-10-15T15:14:20.469 INFO:teuthology.orchestra.run.smithi031.stderr: 'use_existing': True, 2023-10-15T15:14:20.469 INFO:teuthology.orchestra.run.smithi031.stderr: 'user': 'ubuntu'} 2023-10-15T15:14:20.489 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-10-15T15:14:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:14:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:14:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2023-10-15T15:14:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:14:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-15T15:14:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:14:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:14:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2023-10-15T15:14:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:14:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-15T15:14:20.499 INFO:teuthology.orchestra.run.smithi031.stderr:15:14:20 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-15T15:15:20.560 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-15T15:15:20.585 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.585 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-15T15:15:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.617 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-10-15T15:15:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-10-15T15:15:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.621 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.622 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2023-10-15T15:15:20.642 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.643 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi031.front.sepia.ceph.com ; done 2023-10-15T15:15:20.647 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.647 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.647 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-10-15T15:15:20.792 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - WARNING - cbt - sudo killall -9 rados 2023-10-15T15:15:20.792 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - WARNING - cbt - error 1 seen, continuing anyway... 2023-10-15T15:15:20.814 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2023-10-15T15:15:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-10-15T15:15:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-10-15T15:15:20.834 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.835 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2023-10-15T15:15:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.837 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-15T15:15:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.839 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.840 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2023-10-15T15:15:20.842 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-15T15:15:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-15T15:15:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:20.843 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-10-15T15:15:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:21 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:21.150 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:21 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:21.150 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-10-15T15:15:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:21 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:21 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-10-15T15:15:22.870 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:22 - INFO - cbt - Checking Health after pool creation. 2023-10-15T15:15:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:27 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:27 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:27.872 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-15T15:15:28.194 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:28 - INFO - cbt - Final Pool Health Check. 2023-10-15T15:15:33.200 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.200 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.200 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-15T15:15:33.514 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.515 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-15T15:15:33.525 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.525 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-15T15:15:33.552 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.552 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-10-15T15:15:33.554 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.554 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-10-15T15:15:33.555 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.555 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.556 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-15T15:15:33.824 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:33.824 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-15T15:15:34.425 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:34.425 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:34.425 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-10-15T15:15:34.733 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:34.734 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:34.734 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2023-10-15T15:15:34.787 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:34.787 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:34.787 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2023-10-15T15:15:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.538 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2023-10-15T15:15:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-15T15:15:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2023-10-15T15:15:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-15T15:15:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-15T15:15:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - INFO - cbt - Running radosbench write test. 2023-10-15T15:15:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2023-10-15T15:15:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:15:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:15:15:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2023-10-15T15:16:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-15T15:16:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-15T15:16:36.736 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:36.736 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2023-10-15T15:16:36.737 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:36.737 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2023-10-15T15:16:36.739 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:16:36.739 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:36.739 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2023-10-15T15:16:37.048 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2023-10-15T15:16:37.081 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-10-15T15:16:37.082 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi031.front.sepia.ceph.com ; done 2023-10-15T15:16:37.112 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - INFO - cbt - Convert results to json format. 2023-10-15T15:16:37.113 INFO:teuthology.orchestra.run.smithi031.stderr:15:16:37 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:16:37.133 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-10-15T15:16:37.143 DEBUG:teuthology.orchestra.run.smithi031:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-10-15T15:16:37.327 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-10-15T15:16:37.328 INFO:tasks.cbt_performance:reading cbt preferences 2023-10-15T15:16:37.328 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-10-15T15:16:37.389 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi031.front.sepia.ceph.com 2023-10-15T15:16:37.389 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi031.front.sepia.ceph.com 2023-10-15T15:16:37.390 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:16:37.390 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi031.front.sepia.ceph.com of=/dev/stdout 2023-10-15T15:16:37.417 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-15T15:16:37.417 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-10-15T15:16:37.472 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-15T15:16:37.473 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-15T15:16:37.473 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-15T15:16:37.474 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi031.front.sepia.ceph.com of=/dev/stdout 2023-10-15T15:16:37.528 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-15T15:16:37.529 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-10-15T15:16:37.583 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-15T15:16:37.583 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-15T15:16:37.592 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-15T15:16:37.599 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-15T15:16:37.599 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-15T15:16:37.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-15T15:16:37.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:16:37.742+0000 7f8df819e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:16:37.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-15T15:16:37.750+0000 7f8df819e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:16:37.958 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-15T15:16:37.959 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-15T15:16:37.959 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-15T15:16:37.960 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:16:42.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-15T15:16:44.061 INFO:tasks.ceph.osd.0:Stopped 2023-10-15T15:16:44.078 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-15T15:16:44.079 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:16:48.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-15T15:16:48.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-15T15:16:50.181 INFO:tasks.ceph.osd.1:Stopped 2023-10-15T15:16:50.182 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-15T15:16:50.182 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:16:53.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-15T15:16:53.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-15T15:16:56.284 INFO:tasks.ceph.osd.2:Stopped 2023-10-15T15:16:56.285 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-15T15:16:56.285 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-15T15:16:56.286 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:16:56.318 INFO:tasks.ceph.mgr.x:Stopped 2023-10-15T15:16:56.318 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-15T15:16:56.319 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-15T15:16:56.319 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:16:56.371 INFO:tasks.ceph.mon.a:Stopped 2023-10-15T15:16:56.372 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-15T15:16:56.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:16:56.458 INFO:teuthology.orchestra.run.smithi031.stdout:2023-10-15T15:16:05.915473+0000 mon.a (mon.0) 151 : cluster [WRN] Health check failed: Degraded data redundancy: 16/1448 objects degraded (1.105%), 9 pgs degraded (PG_DEGRADED) 2023-10-15T15:16:56.458 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-15T15:16:56.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:16:56.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:16:56.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:16:56.642 INFO:teuthology.orchestra.run.smithi031.stdout:2023-10-15T15:16:05.915473+0000 mon.a (mon.0) 151 : cluster [WRN] Health check failed: Degraded data redundancy: 16/1448 objects degraded (1.105%), 9 pgs degraded (PG_DEGRADED) 2023-10-15T15:16:56.642 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:16:56.642 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-15T15:16:57.009 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:16:57.010 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-15T15:16:57.090 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:16:57.091 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-15T15:16:57.148 INFO:tasks.ceph:Archiving mon data... 2023-10-15T15:16:57.150 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412/data/mon.a.tgz 2023-10-15T15:16:57.150 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-10-15T15:16:57.178 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.uaYOxBjAUp 2023-10-15T15:16:57.179 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.uaYOxBjAUp 2023-10-15T15:16:57.306 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.uaYOxBjAUp 2023-10-15T15:16:57.398 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.uaYOxBjAUp is 176KB 2023-10-15T15:16:57.451 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.uaYOxBjAUp 2023-10-15T15:16:57.466 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-15T15:16:57.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-15T15:16:57.535 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-15T15:16:57.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-15T15:16:57.620 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-15T15:16:57.624 INFO:tasks.ceph:Archiving crash dumps... 2023-10-15T15:16:57.626 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412/remote/smithi031/crash 2023-10-15T15:16:57.627 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-15T15:16:57.694 INFO:tasks.ceph:Compressing logs... 2023-10-15T15:16:57.695 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-15T15:16:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-15T15:16:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83022.log 2023-10-15T15:16:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82252.log 2023-10-15T15:16:57.777 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.83022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83124.log 2023-10-15T15:16:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79429.log 2023-10-15T15:16:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82123.log 2023-10-15T15:16:57.778 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83124.log: gzip -5 --verbose/var/log/ceph/ceph.tmp-client.admin.79429.log: -- /var/log/ceph/ceph-client.admin.80998.log 2023-10-15T15:16:57.785 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81771.log 2023-10-15T15:16:57.785 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82123.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.80998.log: /var/log/ceph/ceph-client.admin.81266.log 2023-10-15T15:16:57.785 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83085.log 2023-10-15T15:16:57.785 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81916.log 2023-10-15T15:16:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82294.log 2023-10-15T15:16:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83085.log: gzip/var/log/ceph/ceph-client.admin.81916.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82210.log 2023-10-15T15:16:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82336.log 2023-10-15T15:16:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81673.log 2023-10-15T15:16:57.786 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81236.log 2023-10-15T15:16:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82336.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-15T15:16:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81673.log: gzip/var/log/ceph/ceph-client.admin.81236.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81998.log 2023-10-15T15:16:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81038.log 2023-10-15T15:16:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-15T15:16:57.787 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81996.log 2023-10-15T15:16:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82378.log 2023-10-15T15:16:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83173.log 2023-10-15T15:16:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83237.log 2023-10-15T15:16:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81832.log 2023-10-15T15:16:57.788 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.81996.log: /var/log/ceph/ceph-client.admin.82378.log: /var/log/ceph/ceph-client.admin.83173.log: /var/log/ceph/ceph-client.admin.83237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80800.log 2023-10-15T15:16:57.788 INFO:teuthology.orchestra.run.smithi031.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.83237.log.gz 2023-10-15T15:16:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83238.log 2023-10-15T15:16:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81832.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.80800.log: /var/log/ceph/ceph-client.admin.83149.log 2023-10-15T15:16:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83732.log 2023-10-15T15:16:57.789 INFO:teuthology.orchestra.run.smithi031.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.83238.log.gz 2023-10-15T15:16:57.789 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82124.log 2023-10-15T15:16:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.83149.log: /var/log/ceph/ceph-client.admin.83732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81729.log 2023-10-15T15:16:57.790 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83732.log.gz 2023-10-15T15:16:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82124.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-15T15:16:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81997.log 2023-10-15T15:16:57.790 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-15T15:16:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82122.log 2023-10-15T15:16:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81078.log 2023-10-15T15:16:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82122.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-15T15:16:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81078.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-15T15:16:57.791 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80888.log 2023-10-15T15:16:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81874.log 2023-10-15T15:16:57.792 INFO:teuthology.orchestra.run.smithi031.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-15T15:16:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.80888.log: 66.4% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79429.log.gz 2023-10-15T15:16:57.792 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81874.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81236.log.gz 2023-10-15T15:16:57.793 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81266.log.gz 2023-10-15T15:16:57.794 INFO:teuthology.orchestra.run.smithi031.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-15T15:16:57.797 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80998.log.gz 2023-10-15T15:16:57.797 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81038.log.gz 2023-10-15T15:16:57.801 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81673.log.gz 2023-10-15T15:16:57.802 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81771.log.gz 2023-10-15T15:16:57.802 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81916.log.gz 2023-10-15T15:16:57.802 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81998.log.gz 2023-10-15T15:16:57.802 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82123.log.gz 2023-10-15T15:16:57.802 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82210.log.gz 2023-10-15T15:16:57.803 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82252.log.gz 2023-10-15T15:16:57.803 INFO:teuthology.orchestra.run.smithi031.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-15T15:16:57.803 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82294.log.gz 2023-10-15T15:16:57.803 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82336.log.gz 2023-10-15T15:16:57.805 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83022.log.gz 2023-10-15T15:16:57.805 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83085.log.gz 2023-10-15T15:16:57.805 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83124.log.gz 2023-10-15T15:16:57.815 INFO:teuthology.orchestra.run.smithi031.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.80800.log.gz 2023-10-15T15:16:57.816 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80888.log.gz 2023-10-15T15:16:57.816 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81996.log.gz 2023-10-15T15:16:57.816 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81997.log.gz 2023-10-15T15:16:57.816 INFO:teuthology.orchestra.run.smithi031.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.82124.log.gz 2023-10-15T15:16:57.816 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82122.log.gz 2023-10-15T15:16:57.817 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82378.log.gz 2023-10-15T15:16:57.817 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83149.log.gz 2023-10-15T15:16:57.817 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83173.log.gz 2023-10-15T15:16:57.817 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81874.log.gz 2023-10-15T15:16:57.817 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81832.log.gz 2023-10-15T15:16:57.818 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81729.log.gz 2023-10-15T15:16:57.823 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81078.log.gz 2023-10-15T15:16:57.918 INFO:teuthology.orchestra.run.smithi031.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-15T15:16:58.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-15T15:16:59.295 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-15T15:16:59.504 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-15T15:16:59.552 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-15T15:16:59.555 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-15T15:16:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m1.805s 2023-10-15T15:16:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m3.766s 2023-10-15T15:16:59.556 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.191s 2023-10-15T15:16:59.556 INFO:tasks.ceph:Archiving logs... 2023-10-15T15:16:59.557 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412/remote/smithi031/log 2023-10-15T15:16:59.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-15T15:17:00.198 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-15T15:17:00.210 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-15T15:17:00.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-15T15:17:00.269 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-15T15:17:00.277 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-15T15:17:00.277 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-15T15:17:00.296 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2023-10-15T15:17:00.367 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-15T15:17:00.367 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-10-15T15:17:00.367 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +1508us[+1519us] +/- 76ms 2023-10-15T15:17:00.368 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -850us[ -838us] +/- 67ms 2023-10-15T15:17:00.368 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -599us[ -588us] +/- 87ms 2023-10-15T15:17:00.368 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:17:00.376 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-15T15:17:00.386 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-15T15:17:00.387 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-15T15:17:00.397 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-15T15:17:00.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-15T15:17:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:17:00.612 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-15T15:17:00.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-15T15:17:00.630 INFO:teuthology.task.internal:Duration was 740.064724 seconds 2023-10-15T15:17:00.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-15T15:17:00.640 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-15T15:17:00.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-15T15:17:00.802 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-15T15:17:01.280 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-15T15:17:01.280 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-10-15T15:17:01.281 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-15T15:17:01.303 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-15T15:17:01.303 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-15T15:17:01.376 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-15T15:17:01.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-15T15:17:01.522 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-15T15:17:01.532 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-15T15:17:01.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-15T15:17:01.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-15T15:17:01.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-10-15T15:17:01.725 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-15T15:17:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:17:01.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-15T15:17:01.790 INFO:teuthology.task.internal:Transferring archived files... 2023-10-15T15:17:01.791 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412/remote/smithi031 2023-10-15T15:17:01.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-15T15:17:01.884 INFO:teuthology.task.internal:Removing archive directory... 2023-10-15T15:17:01.885 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-15T15:17:01.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-15T15:17:01.949 INFO:teuthology.task.internal:Not uploading archives. 2023-10-15T15:17:01.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-15T15:17:01.958 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-15T15:17:01.959 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-15T15:17:01.989 INFO:teuthology.orchestra.run.smithi031.stdout: 265463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 15 15:17 /home/ubuntu/cephtest 2023-10-15T15:17:01.990 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-15T15:17:02.000 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-15T15:17:02.020 INFO:teuthology.nuke:Checking targets against current locks 2023-10-15T15:17:02.047 DEBUG:teuthology.nuke:shortname: smithi031 2023-10-15T15:17:02.048 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-15T15:17:02.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428412', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-15 14:57:15.944355', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNjKeyBa2FxzpICfsMHddHjFSUJ1pcYtdW7Uy1MaBInIIOBsYRVwu/Kr6EW0b0LVx+F9oYFsJUxHq9lx8POqc='} 2023-10-15T15:17:02.119 INFO:teuthology.orchestra.console.smithi031:Power off 2023-10-15T15:17:02.119 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-15T15:17:02.143 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-10-15T15:17:02.144 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:17:02.165 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-15T15:17:06.166 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:17:06.275 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-15T15:17:10.279 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:17:10.387 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-15T15:17:14.390 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:17:14.498 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-15T15:17:14.599 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-10-15T15:17:14.704 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=main&var-job_id_selected=7428412 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 740.0647239685059 failure_reason: '"2023-10-15T15:16:05.915473+0000 mon.a (mon.0) 151 : cluster [WRN] Health check failed: Degraded data redundancy: 16/1448 objects degraded (1.105%), 9 pgs degraded (PG_DEGRADED)" in cluster log' flavor: crimson owner: scheduled_matan@teuthology success: false 2023-10-15T15:17:14.704 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-15T15:17:14.780 INFO:teuthology.run:FAIL