2023-12-03T10:53:27.671 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-03T10:53:27.672 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_matan@teuthology --archive /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096 --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} --name matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi -- /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096/orig.config.yaml 2023-12-03T10:53:27.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T10:53:27.800 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096 branch: wip-matanb-crimson-snaptrimevent-lifetime-v3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: null first_in_suite: false job_id: '7476096' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-snaptrimevent-lifetime-v3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 6f1c8e8f510596dfa1fb82232bce9b991071a48d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 6f1c8e8f510596dfa1fb82232bce9b991071a48d workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 67b75549b33f045a83f5bf55b677d8842cb7bac4 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9570 sha1: 6f1c8e8f510596dfa1fb82232bce9b991071a48d sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_67b75549b33f045a83f5bf55b677d8842cb7bac4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 67b75549b33f045a83f5bf55b677d8842cb7bac4 targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlcorkTQv+ZPzLMQe2htM82ypmH64T70NGyY4IZbj7rBEl+FREQYPcmF0GsnyUk/yCA1smrnANdYuh1Yr5mqZ8= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-03_10:46:10 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-03T10:53:27.800 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_67b75549b33f045a83f5bf55b677d8842cb7bac4/qa; will attempt to use it 2023-12-03T10:53:27.800 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_67b75549b33f045a83f5bf55b677d8842cb7bac4/qa/tasks 2023-12-03T10:53:27.801 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-03T10:53:27.802 INFO:teuthology.task.internal:Checking packages... 2023-12-03T10:53:27.825 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '6f1c8e8f510596dfa1fb82232bce9b991071a48d' 2023-12-03T10:53:27.825 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-03T10:53:27.825 INFO:teuthology.packaging:ref: None 2023-12-03T10:53:27.825 INFO:teuthology.packaging:tag: None 2023-12-03T10:53:27.825 INFO:teuthology.packaging:branch: wip-matanb-crimson-snaptrimevent-lifetime-v3 2023-12-03T10:53:27.825 INFO:teuthology.packaging:sha1: 6f1c8e8f510596dfa1fb82232bce9b991071a48d 2023-12-03T10:53:27.825 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-snaptrimevent-lifetime-v3 2023-12-03T10:53:28.147 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7560.g6f1c8e8f 2023-12-03T10:53:28.148 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-03T10:53:28.155 INFO:teuthology.task.internal:no buildpackages task found 2023-12-03T10:53:28.155 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-03T10:53:28.195 INFO:teuthology.task.internal:Saving configuration 2023-12-03T10:53:28.210 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-03T10:53:28.219 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T10:53:28.245 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 10:48:44.408954', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlcorkTQv+ZPzLMQe2htM82ypmH64T70NGyY4IZbj7rBEl+FREQYPcmF0GsnyUk/yCA1smrnANdYuh1Yr5mqZ8='} 2023-12-03T10:53:28.245 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-03T10:53:28.251 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-03T10:53:28.251 INFO:teuthology.run_tasks:Running task console_log... 2023-12-03T10:53:28.302 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0b55cf09d0>, signals=[15]) 2023-12-03T10:53:28.302 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-03T10:53:28.308 INFO:teuthology.task.internal:Opening connections... 2023-12-03T10:53:28.308 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T10:53:28.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T10:53:28.406 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-03T10:53:28.413 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2023-12-03T10:53:28.475 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2023-12-03T10:53:28.475 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T10:53:28.531 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T10:53:28.532 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2023-12-03T10:53:28.554 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-03T10:53:28.562 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-03T10:53:28.603 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-03T10:53:28.603 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2023-12-03T10:53:28.619 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-03T10:53:28.627 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-03T10:53:28.627 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2023-12-03T10:53:28.682 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-03T10:53:28.713 INFO:teuthology.run_tasks:Running task kernel... 2023-12-03T10:53:28.727 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-03T10:53:28.727 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-03T10:53:28.727 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T10:53:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T10:53:28.743 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2023-12-03T10:53:28.800 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2023-12-03T10:53:28.800 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-361.el9.x86_64 2023-12-03T10:53:28.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2023-12-03T10:53:31.126 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.4 MB 00:00 2023-12-03T10:53:32.560 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2023-12-03T10:53:34.392 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 7.9 MB 00:01 2023-12-03T10:53:37.018 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 12 MB/s | 18 MB 00:01 2023-12-03T10:53:41.410 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 5.8 MB 00:00 2023-12-03T10:53:43.136 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2023-12-03T10:53:55.894 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 1.6 MB/s | 20 MB 00:12 2023-12-03T10:54:00.936 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2023-12-03T10:54:03.384 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T10:54:03.384 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T10:54:03.464 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T10:54:03.471 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T10:54:03.471 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2023-12-03T10:54:03.471 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T10:54:03.471 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T10:54:03.471 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 96 M 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 121 M 2023-12-03T10:54:03.472 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T10:54:05.612 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 2.7 MB/s | 5.6 MB 00:02 2023-12-03T10:54:13.695 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 2.0 MB/s | 20 MB 00:10 2023-12-03T10:54:17.428 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-core-5.14.0-388.el9.x86_6 2.7 MB/s | 32 MB 00:11 2023-12-03T10:54:17.761 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 2.7 MB/s | 38 MB 00:14 2023-12-03T10:54:17.762 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T10:54:17.762 INFO:teuthology.orchestra.run.smithi019.stdout:Total 6.7 MB/s | 96 MB 00:14 2023-12-03T10:54:18.659 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T10:54:19.106 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T10:54:19.106 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T10:54:20.473 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T10:54:20.473 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T10:54:23.251 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T10:54:23.929 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T10:54:24.063 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T10:54:26.973 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T10:54:27.147 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T10:54:32.137 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T10:54:32.281 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T10:54:37.301 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T10:55:08.049 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T10:55:08.094 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T10:55:08.500 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T10:55:08.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T10:55:08.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T10:55:08.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T10:55:08.906 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T10:55:09.452 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T10:55:09.599 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T10:55:09.600 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T10:55:09.600 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T10:55:09.600 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T10:55:09.600 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T10:55:09.601 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-03T10:55:09.601 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T10:55:09.601 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T10:55:09.723 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T10:55:09.723 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T10:55:09.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2023-12-03T10:55:09.764 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2023-12-03T10:55:09.764 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2023-12-03T10:55:09.764 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2023-12-03T10:55:09.765 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T10:55:09.766 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T10:55:09.766 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T10:55:09.766 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T10:55:09.766 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2023-12-03T10:55:09.766 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T10:55:09.766 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T10:55:09.767 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T10:55:09.767 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-388.el9.x86_64 2023-12-03T10:55:09.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T10:55:10.402 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2023-12-03T10:55:11.850 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T10:55:11.889 INFO:teuthology.orchestra.run.smithi019.stderr:done 2023-12-03T10:55:11.892 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2023-12-03T10:55:11.906 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.1epSRI39Pm 2023-12-03T10:55:11.907 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1epSRI39Pm 2023-12-03T10:55:11.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.1epSRI39Pm 2023-12-03T10:55:12.117 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.1epSRI39Pm is 6KB 2023-12-03T10:55:12.136 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.1epSRI39Pm 2023-12-03T10:55:12.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T10:55:12.219 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T10:55:12.219 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T10:55:12.219 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T10:55:12.220 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T10:55:12.221 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T10:55:12.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2023-12-03T10:55:42.403 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-03T10:55:42.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-12-03T10:55:42.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T10:56:00.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2023-12-03T10:56:09.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-12-03T10:56:09.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T10:57:09.940 DEBUG:teuthology.orchestra.remote:timed out 2023-12-03T10:57:21.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-12-03T10:57:21.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T10:57:22.302 DEBUG:teuthology.orchestra.run.smithi019:> true 2023-12-03T10:57:22.677 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-12-03T10:57:22.678 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-03T10:57:22.678 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2023-12-03T10:57:22.693 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-388.el9.x86_64 2023-12-03T10:57:22.693 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T10:57:22.693 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T10:57:22.693 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T10:57:23.694 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-03T10:57:23.695 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T10:57:23.759 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2023-12-03T10:57:23.795 DEBUG:teuthology.parallel:result is None 2023-12-03T10:57:23.796 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-03T10:57:23.805 INFO:teuthology.task.internal:Creating test directory... 2023-12-03T10:57:23.805 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-03T10:57:23.823 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-03T10:57:23.843 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-03T10:57:23.893 INFO:teuthology.task.internal:Creating archive directory... 2023-12-03T10:57:23.893 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-03T10:57:23.924 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-03T10:57:23.943 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-03T10:57:23.943 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-03T10:57:23.997 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T10:57:24.015 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T10:57:24.016 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-03T10:57:24.023 INFO:teuthology.task.internal:Configuring sudo... 2023-12-03T10:57:24.023 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-03T10:57:24.090 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-03T10:57:24.100 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-03T10:57:24.100 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-03T10:57:24.147 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T10:57:24.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T10:57:24.286 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T10:57:24.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T10:57:24.416 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T10:57:24.417 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-03T10:57:24.488 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2023-12-03T10:57:24.588 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T10:57:24.983 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-03T10:57:24.991 INFO:teuthology.task.internal:Starting timer... 2023-12-03T10:57:24.992 INFO:teuthology.run_tasks:Running task pcp... 2023-12-03T10:57:25.037 INFO:teuthology.run_tasks:Running task selinux... 2023-12-03T10:57:25.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2023-12-03T10:57:25.140 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2023-12-03T10:57:25.142 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-03T10:57:25.142 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2023-12-03T10:57:25.172 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2023-12-03T10:57:25.172 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive'} 2023-12-03T10:57:25.172 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T10:57:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T10:57:25.244 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-03T10:57:25.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2023-12-03T10:57:25.324 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-03T10:57:25.333 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-03T10:57:25.343 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-03T10:57:25.540 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-03T10:57:25.558 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-03T10:57:25.560 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-03T11:03:25.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com')] 2023-12-03T11:03:25.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-12-03T11:03:25.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:03:25.937 DEBUG:teuthology.orchestra.run.smithi019:> true 2023-12-03T11:03:26.010 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-12-03T11:03:26.011 INFO:teuthology.run_tasks:Running task clock... 2023-12-03T11:03:26.021 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-03T11:03:26.021 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-03T11:03:26.022 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T11:03:26.086 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-03T11:03:26.107 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-03T11:03:26.143 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2023-12-03T11:03:26.160 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2023-12-03T11:03:26.180 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-03T11:03:26.200 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-03T11:03:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2023-12-03T11:03:26.250 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T11:03:26.250 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2023-12-03T11:03:26.250 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:03:26.250 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:03:26.250 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:03:26.251 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:03:26.251 INFO:teuthology.run_tasks:Running task install... 2023-12-03T11:03:26.260 DEBUG:teuthology.task.install:project ceph 2023-12-03T11:03:26.260 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d'}} 2023-12-03T11:03:26.260 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d'} 2023-12-03T11:03:26.260 INFO:teuthology.task.install:Using flavor: crimson 2023-12-03T11:03:26.266 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-03T11:03:26.266 INFO:teuthology.task.install:extra packages: [] 2023-12-03T11:03:26.267 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d', 'tag': None, 'wait_for_package': False} 2023-12-03T11:03:26.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f1c8e8f510596dfa1fb82232bce9b991071a48d 2023-12-03T11:03:26.412 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v3/6f1c8e8f510596dfa1fb82232bce9b991071a48d/centos/9/flavors/crimson/ 2023-12-03T11:03:26.412 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7560.g6f1c8e8f 2023-12-03T11:03:26.787 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v3/6f1c8e8f510596dfa1fb82232bce9b991071a48d/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v3/6f1c8e8f510596dfa1fb82232bce9b991071a48d/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v3/6f1c8e8f510596dfa1fb82232bce9b991071a48d/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-03T11:03:26.788 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:03:26.788 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-03T11:03:26.821 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-03T11:03:26.821 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f1c8e8f510596dfa1fb82232bce9b991071a48d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-03T11:03:26.894 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-03T11:03:26.984 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-03T11:03:27.014 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2023-12-03T11:03:27.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2023-12-03T11:03:27.270 INFO:teuthology.orchestra.run.smithi019.stdout:73 files removed 2023-12-03T11:03:27.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2023-12-03T11:03:28.505 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 93 kB/s | 81 kB 00:00 2023-12-03T11:03:29.180 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 15 kB/s | 9.5 kB 00:00 2023-12-03T11:03:29.948 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2023-12-03T11:03:30.366 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-03T11:03:31.767 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2023-12-03T11:03:33.799 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 7.9 MB 00:01 2023-12-03T11:03:35.800 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-12-03T11:03:40.568 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 5.8 MB 00:01 2023-12-03T11:03:41.944 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 15 kB 00:00 2023-12-03T11:03:44.106 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 20 MB 00:02 2023-12-03T11:03:49.062 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-03T11:03:51.924 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:03:51.935 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:03:51.935 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:03:51.935 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:03:51.935 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:03:51.935 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 8.4 M 2023-12-03T11:03:51.935 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 3.1 M 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 3.0 M 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 6.6 M 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 17 M 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 24 k 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 729 k 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-03T11:03:51.936 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 470 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 4.5 M 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 45 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 129 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 162 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 321 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 299 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 99 k 2023-12-03T11:03:51.937 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout:Install 27 Packages 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 54 M 2023-12-03T11:03:51.938 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:03:58.854 INFO:teuthology.orchestra.run.smithi019.stdout:(1/29): ceph-base-18.0.0-7560.g6f1c8e8f.el9.x86 988 kB/s | 6.6 MB 00:06 2023-12-03T11:03:58.971 INFO:teuthology.orchestra.run.smithi019.stdout:(2/29): ceph-selinux-18.0.0-7560.g6f1c8e8f.el9. 206 kB/s | 24 kB 00:00 2023-12-03T11:04:00.022 INFO:teuthology.orchestra.run.smithi019.stdout:(3/29): libcephfs2-18.0.0-7560.g6f1c8e8f.el9.x8 694 kB/s | 729 kB 00:01 2023-12-03T11:04:00.688 INFO:teuthology.orchestra.run.smithi019.stdout:(4/29): libradosstriper1-18.0.0-7560.g6f1c8e8f. 706 kB/s | 470 kB 00:00 2023-12-03T11:04:03.672 INFO:teuthology.orchestra.run.smithi019.stdout:(5/29): ceph-radosgw-18.0.0-7560.g6f1c8e8f.el9. 737 kB/s | 8.4 MB 00:11 2023-12-03T11:04:03.789 INFO:teuthology.orchestra.run.smithi019.stdout:(6/29): python3-ceph-argparse-18.0.0-7560.g6f1c 387 kB/s | 45 kB 00:00 2023-12-03T11:04:04.089 INFO:teuthology.orchestra.run.smithi019.stdout:(7/29): python3-ceph-common-18.0.0-7560.g6f1c8e 430 kB/s | 129 kB 00:00 2023-12-03T11:04:04.472 INFO:teuthology.orchestra.run.smithi019.stdout:(8/29): python3-cephfs-18.0.0-7560.g6f1c8e8f.el 423 kB/s | 162 kB 00:00 2023-12-03T11:04:05.064 INFO:teuthology.orchestra.run.smithi019.stdout:(9/29): python3-rados-18.0.0-7560.g6f1c8e8f.el9 543 kB/s | 321 kB 00:00 2023-12-03T11:04:05.515 INFO:teuthology.orchestra.run.smithi019.stdout:(10/29): librgw2-18.0.0-7560.g6f1c8e8f.el9.x86_ 954 kB/s | 4.5 MB 00:04 2023-12-03T11:04:05.598 INFO:teuthology.orchestra.run.smithi019.stdout:(11/29): python3-rbd-18.0.0-7560.g6f1c8e8f.el9. 560 kB/s | 299 kB 00:00 2023-12-03T11:04:05.623 INFO:teuthology.orchestra.run.smithi019.stdout:(12/29): python3-rgw-18.0.0-7560.g6f1c8e8f.el9. 918 kB/s | 99 kB 00:00 2023-12-03T11:04:05.749 INFO:teuthology.orchestra.run.smithi019.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 703 kB/s | 106 kB 00:00 2023-12-03T11:04:05.774 INFO:teuthology.orchestra.run.smithi019.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-12-03T11:04:05.799 INFO:teuthology.orchestra.run.smithi019.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2023-12-03T11:04:05.883 INFO:teuthology.orchestra.run.smithi019.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2023-12-03T11:04:05.908 INFO:teuthology.orchestra.run.smithi019.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2023-12-03T11:04:05.934 INFO:teuthology.orchestra.run.smithi019.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2023-12-03T11:04:06.126 INFO:teuthology.orchestra.run.smithi019.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2023-12-03T11:04:06.168 INFO:teuthology.orchestra.run.smithi019.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 587 kB/s | 24 kB 00:00 2023-12-03T11:04:06.218 INFO:teuthology.orchestra.run.smithi019.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 975 kB/s | 49 kB 00:00 2023-12-03T11:04:06.268 INFO:teuthology.orchestra.run.smithi019.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-12-03T11:04:06.368 INFO:teuthology.orchestra.run.smithi019.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2023-12-03T11:04:06.410 INFO:teuthology.orchestra.run.smithi019.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2023-12-03T11:04:06.461 INFO:teuthology.orchestra.run.smithi019.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 4.3 MB/s | 837 kB 00:00 2023-12-03T11:04:06.503 INFO:teuthology.orchestra.run.smithi019.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2023-12-03T11:04:07.336 INFO:teuthology.orchestra.run.smithi019.stdout:(27/29): ceph-common-18.0.0-7560.g6f1c8e8f.el9. 1.1 MB/s | 17 MB 00:15 2023-12-03T11:04:10.003 INFO:teuthology.orchestra.run.smithi019.stdout:(28/29): librados2-18.0.0-7560.g6f1c8e8f.el9.x8 898 kB/s | 3.1 MB 00:03 2023-12-03T11:04:10.903 INFO:teuthology.orchestra.run.smithi019.stdout:(29/29): librbd1-18.0.0-7560.g6f1c8e8f.el9.x86_ 703 kB/s | 3.0 MB 00:04 2023-12-03T11:04:10.908 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:04:10.908 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.8 MB/s | 54 MB 00:18 2023-12-03T11:04:10.908 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:04:10.949 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:04:10.949 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:04:11.425 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:04:11.426 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:04:12.808 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:04:13.152 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-03T11:04:13.230 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/31 2023-12-03T11:04:13.306 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/31 2023-12-03T11:04:13.397 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 3/31 2023-12-03T11:04:13.424 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 4/31 2023-12-03T11:04:13.490 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 4/31 2023-12-03T11:04:13.519 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-12-03T11:04:13.544 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-12-03T11:04:13.578 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-12-03T11:04:13.622 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:18.0.0-7560.g6f1c8e8f.el9. 8/31 2023-12-03T11:04:13.690 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 9/31 2023-12-03T11:04:13.713 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:18.0.0-7560.g6f1c8e8f.el9.x86_6 10/31 2023-12-03T11:04:13.802 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7560.g6f1c8e8f.el9.x86_6 10/31 2023-12-03T11:04:13.837 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-12-03T11:04:13.948 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-03T11:04:13.980 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-03T11:04:14.013 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-03T11:04:14.053 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-03T11:04:14.082 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-03T11:04:14.115 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-03T11:04:14.449 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-03T11:04:14.478 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 19/31 2023-12-03T11:04:14.560 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 19/31 2023-12-03T11:04:14.591 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 20/31 2023-12-03T11:04:14.627 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-03T11:04:15.080 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-03T11:04:15.105 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 23/31 2023-12-03T11:04:15.177 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 23/31 2023-12-03T11:04:15.247 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 24/31 2023-12-03T11:04:15.283 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x8 25/31 2023-12-03T11:04:17.435 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 26/31 2023-12-03T11:04:17.471 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 26/31 2023-12-03T11:04:18.393 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 26/31 2023-12-03T11:04:18.441 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 27/31 2023-12-03T11:04:18.540 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 27/31 2023-12-03T11:04:18.541 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-03T11:04:18.541 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-03T11:04:18.541 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:18.575 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 28/31 2023-12-03T11:04:28.220 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 28/31 2023-12-03T11:04:28.220 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2023-12-03T11:04:28.220 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2023-12-03T11:04:28.220 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2023-12-03T11:04:28.221 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2023-12-03T11:04:28.221 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2023-12-03T11:04:28.221 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2023-12-03T11:04:28.221 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2023-12-03T11:04:28.221 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:29.081 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 29/31 2023-12-03T11:04:29.107 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 29/31 2023-12-03T11:04:29.107 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:04:29.107 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-03T11:04:29.107 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:04:29.107 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:04:29.107 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:29.109 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-03T11:04:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-03T11:04:29.175 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 3/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 4/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 5/31 2023-12-03T11:04:30.379 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:18.0.0-7560.g6f1c8e8f.el9.x86_6 6/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 7/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7560.g6f1c8e8f.el9. 8/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x8 9/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 10/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 11/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 12/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 13/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-03T11:04:30.380 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 28/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-03T11:04:30.381 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 30/31 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.300 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.301 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:31.302 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:04:31.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2023-12-03T11:04:32.229 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:43 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:04:32.381 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 118 M 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 158 k 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-12-03T11:04:32.382 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout:Install 10 Packages 2023-12-03T11:04:32.383 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:04:32.384 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 121 M 2023-12-03T11:04:32.384 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.1 G 2023-12-03T11:04:32.384 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:04:32.606 INFO:teuthology.orchestra.run.smithi019.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-03T11:04:32.656 INFO:teuthology.orchestra.run.smithi019.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 8.2 MB/s | 418 kB 00:00 2023-12-03T11:04:32.698 INFO:teuthology.orchestra.run.smithi019.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.5 MB/s | 186 kB 00:00 2023-12-03T11:04:32.740 INFO:teuthology.orchestra.run.smithi019.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-03T11:04:32.782 INFO:teuthology.orchestra.run.smithi019.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-03T11:04:32.816 INFO:teuthology.orchestra.run.smithi019.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-03T11:04:33.166 INFO:teuthology.orchestra.run.smithi019.stdout:(7/10): libcephsqlite-18.0.0-7560.g6f1c8e8f.el9 217 kB/s | 158 kB 00:00 2023-12-03T11:04:33.399 INFO:teuthology.orchestra.run.smithi019.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.2 MB/s | 1.3 MB 00:00 2023-12-03T11:04:33.491 INFO:teuthology.orchestra.run.smithi019.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 374 kB/s | 122 kB 00:00 2023-12-03T11:07:03.826 INFO:teuthology.orchestra.run.smithi019.stdout:(10/10): ceph-test-18.0.0-7560.g6f1c8e8f.el9.x8 796 kB/s | 118 MB 02:31 2023-12-03T11:07:03.828 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:07:03.828 INFO:teuthology.orchestra.run.smithi019.stdout:Total 815 kB/s | 121 MB 02:31 2023-12-03T11:07:03.828 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:07:03.849 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:07:03.849 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:07:04.715 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:07:04.716 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:07:05.880 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:07:05.967 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-12-03T11:07:06.018 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-12-03T11:07:06.061 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-12-03T11:07:06.109 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-12-03T11:07:06.142 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-12-03T11:07:06.175 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-12-03T11:07:06.210 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-12-03T11:07:06.253 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-12-03T11:07:06.280 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 9/10 2023-12-03T11:07:23.452 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 9/10 2023-12-03T11:07:23.522 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 10/10 2023-12-03T11:07:31.673 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 10/10 2023-12-03T11:07:31.673 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/10 2023-12-03T11:07:31.673 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-12-03T11:07:31.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-16.el9.x86_64 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-12-03T11:07:32.155 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:07:32.156 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:07:32.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2023-12-03T11:07:33.072 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:44 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:07:33.227 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 6.6 k 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 2.0 M 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 1.5 M 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 243 k 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 8.9 M 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 117 M 2023-12-03T11:07:33.230 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-03T11:07:33.231 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-03T11:07:33.232 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout:Install 36 Packages 2023-12-03T11:07:33.233 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:07:33.234 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 134 M 2023-12-03T11:07:33.234 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.4 G 2023-12-03T11:07:33.234 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:07:33.863 INFO:teuthology.orchestra.run.smithi019.stdout:(1/36): ceph-18.0.0-7560.g6f1c8e8f.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2023-12-03T11:07:35.271 INFO:teuthology.orchestra.run.smithi019.stdout:(2/36): ceph-mgr-18.0.0-7560.g6f1c8e8f.el9.x86_ 806 kB/s | 1.5 MB 00:01 2023-12-03T11:07:35.763 INFO:teuthology.orchestra.run.smithi019.stdout:(3/36): ceph-mds-18.0.0-7560.g6f1c8e8f.el9.x86_ 875 kB/s | 2.0 MB 00:02 2023-12-03T11:07:36.022 INFO:teuthology.orchestra.run.smithi019.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7560.g6f1c 940 kB/s | 243 kB 00:00 2023-12-03T11:07:36.139 INFO:teuthology.orchestra.run.smithi019.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-03T11:07:36.180 INFO:teuthology.orchestra.run.smithi019.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-03T11:07:36.214 INFO:teuthology.orchestra.run.smithi019.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-12-03T11:07:36.248 INFO:teuthology.orchestra.run.smithi019.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-03T11:07:36.281 INFO:teuthology.orchestra.run.smithi019.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-03T11:07:36.323 INFO:teuthology.orchestra.run.smithi019.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-03T11:07:36.407 INFO:teuthology.orchestra.run.smithi019.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2023-12-03T11:07:36.457 INFO:teuthology.orchestra.run.smithi019.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2023-12-03T11:07:36.507 INFO:teuthology.orchestra.run.smithi019.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-03T11:07:36.558 INFO:teuthology.orchestra.run.smithi019.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-03T11:07:36.616 INFO:teuthology.orchestra.run.smithi019.stdout:(15/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2023-12-03T11:07:36.700 INFO:teuthology.orchestra.run.smithi019.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2023-12-03T11:07:37.317 INFO:teuthology.orchestra.run.smithi019.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 245 kB/s | 151 kB 00:00 2023-12-03T11:07:37.475 INFO:teuthology.orchestra.run.smithi019.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2023-12-03T11:07:37.784 INFO:teuthology.orchestra.run.smithi019.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 557 kB/s | 172 kB 00:00 2023-12-03T11:07:38.160 INFO:teuthology.orchestra.run.smithi019.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 956 kB/s | 358 kB 00:00 2023-12-03T11:07:38.309 INFO:teuthology.orchestra.run.smithi019.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 71 kB/s | 11 kB 00:00 2023-12-03T11:07:38.468 INFO:teuthology.orchestra.run.smithi019.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 111 kB/s | 18 kB 00:00 2023-12-03T11:07:38.644 INFO:teuthology.orchestra.run.smithi019.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 132 kB/s | 23 kB 00:00 2023-12-03T11:07:38.844 INFO:teuthology.orchestra.run.smithi019.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 97 kB/s | 19 kB 00:00 2023-12-03T11:07:39.002 INFO:teuthology.orchestra.run.smithi019.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 124 kB/s | 20 kB 00:00 2023-12-03T11:07:39.153 INFO:teuthology.orchestra.run.smithi019.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 309 kB/s | 46 kB 00:00 2023-12-03T11:07:39.361 INFO:teuthology.orchestra.run.smithi019.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 378 kB/s | 79 kB 00:00 2023-12-03T11:07:39.803 INFO:teuthology.orchestra.run.smithi019.stdout:(28/36): ceph-mon-18.0.0-7560.g6f1c8e8f.el9.x86 1.5 MB/s | 8.9 MB 00:05 2023-12-03T11:07:39.829 INFO:teuthology.orchestra.run.smithi019.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 583 kB/s | 272 kB 00:00 2023-12-03T11:07:39.971 INFO:teuthology.orchestra.run.smithi019.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 98 kB/s | 16 kB 00:00 2023-12-03T11:07:40.096 INFO:teuthology.orchestra.run.smithi019.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 286 kB/s | 36 kB 00:00 2023-12-03T11:07:40.246 INFO:teuthology.orchestra.run.smithi019.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 215 kB/s | 90 kB 00:00 2023-12-03T11:07:40.355 INFO:teuthology.orchestra.run.smithi019.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 888 kB/s | 230 kB 00:00 2023-12-03T11:07:40.530 INFO:teuthology.orchestra.run.smithi019.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 114 kB/s | 20 kB 00:00 2023-12-03T11:07:40.830 INFO:teuthology.orchestra.run.smithi019.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 732 kB/s | 427 kB 00:00 2023-12-03T11:08:47.107 INFO:teuthology.orchestra.run.smithi019.stdout:(36/36): ceph-osd-18.0.0-7560.g6f1c8e8f.el9.x86 1.6 MB/s | 117 MB 01:11 2023-12-03T11:08:47.110 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:08:47.110 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.8 MB/s | 134 MB 01:13 2023-12-03T11:08:47.120 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:08:47.163 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:08:47.163 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:08:48.235 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:08:48.235 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:08:50.197 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:08:50.251 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-03T11:08:50.284 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-03T11:08:50.312 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-03T11:08:50.385 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-03T11:08:50.428 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-03T11:08:50.460 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-03T11:08:50.555 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-03T11:08:50.649 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-03T11:08:50.697 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-03T11:08:50.779 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-03T11:08:50.832 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-03T11:08:50.869 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-03T11:08:50.919 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-03T11:08:51.050 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-03T11:08:51.267 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-03T11:08:51.323 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-03T11:08:51.362 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-03T11:08:51.396 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-03T11:08:51.426 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-03T11:08:51.505 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-03T11:08:51.536 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-03T11:08:51.570 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-03T11:08:51.611 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-03T11:08:51.653 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-03T11:08:51.696 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-03T11:08:51.789 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-03T11:08:51.827 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-03T11:08:51.902 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-03T11:08:52.051 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-03T11:08:52.182 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-03T11:08:52.351 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7560.g6f1c8e8f.el9. 31/36 2023-12-03T11:08:52.373 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 32/36 2023-12-03T11:08:52.399 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 32/36 2023-12-03T11:08:52.399 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:08:52.399 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-03T11:08:52.400 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:08:52.400 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:08:52.400 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:11.840 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 33/36 2023-12-03T11:09:11.868 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 33/36 2023-12-03T11:09:11.868 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:09:11.868 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-03T11:09:11.868 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:09:11.868 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:09:11.869 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:12.815 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 34/36 2023-12-03T11:09:12.840 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 34/36 2023-12-03T11:09:12.840 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:09:12.840 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-03T11:09:12.840 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:09:12.840 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:09:12.840 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:13.050 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 35/36 2023-12-03T11:09:13.073 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 35/36 2023-12-03T11:09:13.074 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:09:13.074 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-03T11:09:13.074 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:09:13.074 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:09:13.074 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:13.133 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 36/36 2023-12-03T11:09:23.010 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 36/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 3/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 4/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 5/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7560.g6f1c8e8f.el9. 6/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-03T11:09:23.011 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-03T11:09:23.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-03T11:09:23.013 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-03T11:09:23.013 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-03T11:09:23.014 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-03T11:09:24.087 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-03T11:09:24.088 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:24.089 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:24.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2023-12-03T11:09:24.942 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:35 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:25.039 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:09:25.091 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:25.092 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:09:25.092 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:25.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2023-12-03T11:09:25.760 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:36 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:25.910 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:25.910 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:25.910 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:09:25.910 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 775 k 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 775 k 2023-12-03T11:09:25.911 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 788 k 2023-12-03T11:09:25.912 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:26.257 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-18.0.0-7560.g6f1c8e8f.el9.noarch.rpm 2.2 MB/s | 775 kB 00:00 2023-12-03T11:09:26.257 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:26.257 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.2 MB/s | 775 kB 00:00 2023-12-03T11:09:26.258 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:26.263 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:26.263 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:26.270 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:26.270 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:26.346 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:26.826 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:09:26.922 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:09:27.308 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:09:27.642 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:09:27.643 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:27.643 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:27.643 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:27.643 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:27.643 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:27.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2023-12-03T11:09:28.461 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:39 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:28.615 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:28.615 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:28.615 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 143 k 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:09:28.616 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:28.617 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 143 k 2023-12-03T11:09:28.617 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 437 k 2023-12-03T11:09:28.617 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:28.837 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-18.0.0-7560.g6f1c8e 649 kB/s | 143 kB 00:00 2023-12-03T11:09:28.838 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:28.838 INFO:teuthology.orchestra.run.smithi019.stdout:Total 645 kB/s | 143 kB 00:00 2023-12-03T11:09:28.838 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:28.846 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:28.846 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:28.901 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:28.901 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:29.071 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:29.130 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7560.g6f1c8e8f. 1/1 2023-12-03T11:09:29.155 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7560.g6f1c8e8f. 1/1 2023-12-03T11:09:29.155 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:09:29.156 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-03T11:09:29.156 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:30.190 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7560.g6f1c8e8f. 1/1 2023-12-03T11:09:30.190 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:30.191 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:30.191 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:09:30.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:30.191 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:30.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2023-12-03T11:09:31.016 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:42 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:31.115 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:09:31.167 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:31.168 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:09:31.168 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:31.242 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2023-12-03T11:09:31.844 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:42 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:31.997 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 1.8 M 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 28 k 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 15 k 2023-12-03T11:09:31.998 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout:Install 7 Packages 2023-12-03T11:09:31.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:32.000 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.1 M 2023-12-03T11:09:32.000 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 8.7 M 2023-12-03T11:09:32.000 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:32.457 INFO:teuthology.orchestra.run.smithi019.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7560.g6f1 135 kB/s | 28 kB 00:00 2023-12-03T11:09:32.591 INFO:teuthology.orchestra.run.smithi019.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7560.g6f1c 44 kB/s | 15 kB 00:00 2023-12-03T11:09:32.659 INFO:teuthology.orchestra.run.smithi019.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2023-12-03T11:09:32.860 INFO:teuthology.orchestra.run.smithi019.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 154 kB/s | 31 kB 00:00 2023-12-03T11:09:32.977 INFO:teuthology.orchestra.run.smithi019.stdout:(5/7): ceph-mgr-dashboard-18.0.0-7560.g6f1c8e8f 2.4 MB/s | 1.8 MB 00:00 2023-12-03T11:09:32.993 INFO:teuthology.orchestra.run.smithi019.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 102 kB/s | 41 kB 00:00 2023-12-03T11:09:33.227 INFO:teuthology.orchestra.run.smithi019.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 514 kB/s | 188 kB 00:00 2023-12-03T11:09:33.229 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:33.229 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.7 MB/s | 2.1 MB 00:01 2023-12-03T11:09:33.230 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:33.239 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:33.239 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:33.315 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:33.315 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:33.611 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:33.670 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-03T11:09:33.704 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-03T11:09:33.759 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-03T11:09:33.800 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-03T11:09:33.829 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7560.g6f1c8e8f.el9.n 5/7 2023-12-03T11:09:34.073 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7560.g6f1c8e8f.el9. 6/7 2023-12-03T11:09:34.106 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7560.g6f1c8e8f.el9.noarc 7/7 2023-12-03T11:09:34.697 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7560.g6f1c8e8f.el9.noarc 7/7 2023-12-03T11:09:34.697 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7560.g6f1c8e8f.el9. 1/7 2023-12-03T11:09:34.697 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7560.g6f1c8e8f.el9.noarc 2/7 2023-12-03T11:09:34.698 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7560.g6f1c8e8f.el9.n 3/7 2023-12-03T11:09:34.698 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-03T11:09:34.698 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-03T11:09:34.698 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-03T11:09:35.012 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:35.013 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:35.238 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-03T11:09:35.841 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:46 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:35.997 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 7.4 M 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-03T11:09:35.998 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2023-12-03T11:09:35.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:36.000 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 37 M 2023-12-03T11:09:36.000 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 209 M 2023-12-03T11:09:36.000 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:36.400 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 155 kB/s | 32 kB 00:00 2023-12-03T11:09:36.467 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 257 kB/s | 17 kB 00:00 2023-12-03T11:09:36.567 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 341 kB/s | 34 kB 00:00 2023-12-03T11:09:36.868 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): ceph-mgr-diskprediction-local-18.0.0-75 11 MB/s | 7.4 MB 00:00 2023-12-03T11:09:36.918 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2023-12-03T11:09:37.135 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.0 MB/s | 217 kB 00:00 2023-12-03T11:09:37.294 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 6.5 MB/s | 4.7 MB 00:00 2023-12-03T11:09:37.361 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.7 MB/s | 4.8 MB 00:00 2023-12-03T11:09:37.711 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2023-12-03T11:09:37.862 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2023-12-03T11:09:37.886 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 373 kB/s | 195 kB 00:00 2023-12-03T11:09:38.153 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 556 kB/s | 245 kB 00:00 2023-12-03T11:09:38.157 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:38.157 INFO:teuthology.orchestra.run.smithi019.stdout:Total 17 MB/s | 37 MB 00:02 2023-12-03T11:09:38.171 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:38.197 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:38.198 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:38.602 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:38.603 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:39.326 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:39.411 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-03T11:09:39.441 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-03T11:09:39.539 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-03T11:09:39.569 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-03T11:09:39.754 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-03T11:09:39.784 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-03T11:09:39.871 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-03T11:09:40.557 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-03T11:09:40.693 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-03T11:09:41.988 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-03T11:09:43.072 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-03T11:09:43.092 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7560.g6f1c8 12/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7560.g6f1c8 12/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7560.g6f1c8 1/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-03T11:09:45.523 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-03T11:09:45.524 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-03T11:09:45.524 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-03T11:09:45.992 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:45.993 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:46.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2023-12-03T11:09:46.996 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:05:57 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:47.148 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:47.149 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-12-03T11:09:47.149 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 50 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-03T11:09:47.150 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:47.151 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2023-12-03T11:09:47.152 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2023-12-03T11:09:47.152 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:47.583 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 129 kB/s | 24 kB 00:00 2023-12-03T11:09:47.608 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 84 kB/s | 18 kB 00:00 2023-12-03T11:09:47.758 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 996 kB/s | 174 kB 00:00 2023-12-03T11:09:47.800 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 693 kB/s | 132 kB 00:00 2023-12-03T11:09:47.825 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): ceph-mgr-rook-18.0.0-7560.g6f1c8e8f.el9 116 kB/s | 50 kB 00:00 2023-12-03T11:09:47.851 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2023-12-03T11:09:47.868 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 634 kB/s | 43 kB 00:00 2023-12-03T11:09:48.185 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 45 kB/s | 15 kB 00:00 2023-12-03T11:09:48.210 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 84 kB/s | 32 kB 00:00 2023-12-03T11:09:48.335 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 423 kB/s | 197 kB 00:00 2023-12-03T11:09:48.402 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2023-12-03T11:09:48.444 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2023-12-03T11:09:48.786 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2023-12-03T11:09:48.791 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:48.791 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2023-12-03T11:09:48.791 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:48.807 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:48.807 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:48.924 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:48.925 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:49.463 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:49.641 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-03T11:09:49.696 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-03T11:09:49.753 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-03T11:09:49.788 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-03T11:09:49.830 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-03T11:09:49.926 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-03T11:09:50.037 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-03T11:09:50.090 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-03T11:09:50.857 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-03T11:09:50.912 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-03T11:09:50.945 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-03T11:09:50.993 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-03T11:09:51.048 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:18.0.0-7560.g6f1c8e8f.el9.noarch 13/13 2023-12-03T11:09:51.915 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7560.g6f1c8e8f.el9.noarch 13/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-03T11:09:51.916 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-03T11:09:52.297 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-03T11:09:52.298 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-03T11:09:52.298 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-03T11:09:52.298 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:52.298 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:52.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2023-12-03T11:09:53.170 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:04 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:53.323 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:53.324 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-12-03T11:09:53.324 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:09:53.324 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-12-03T11:09:53.324 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:53.324 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 133 k 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2023-12-03T11:09:53.325 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:53.326 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.8 M 2023-12-03T11:09:53.326 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2023-12-03T11:09:53.326 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:53.888 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 596 kB/s | 228 kB 00:00 2023-12-03T11:09:54.013 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7560.g6f1c8e8f.e 261 kB/s | 133 kB 00:00 2023-12-03T11:09:54.079 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2023-12-03T11:09:54.121 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 538 kB/s | 58 kB 00:00 2023-12-03T11:09:54.139 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 868 kB/s | 51 kB 00:00 2023-12-03T11:09:54.222 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.0 MB/s | 5.8 MB 00:00 2023-12-03T11:09:54.224 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:54.224 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.5 MB/s | 6.8 MB 00:00 2023-12-03T11:09:54.224 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:54.234 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:54.234 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:54.330 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:54.330 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:54.649 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:54.818 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-03T11:09:54.874 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-03T11:09:55.439 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-03T11:09:55.529 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-03T11:09:55.589 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-03T11:09:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 6/6 2023-12-03T11:09:56.404 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 6/6 2023-12-03T11:09:56.404 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/6 2023-12-03T11:09:56.404 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-03T11:09:56.404 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-03T11:09:56.404 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-03T11:09:56.404 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-03T11:09:56.727 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:56.728 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:57.005 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2023-12-03T11:09:57.618 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:08 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:09:57.772 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:09:57.772 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 834 k 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2023-12-03T11:09:57.773 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:57.774 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 912 k 2023-12-03T11:09:57.774 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 3.0 M 2023-12-03T11:09:57.774 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:09:57.983 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 377 kB/s | 79 kB 00:00 2023-12-03T11:09:58.066 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-18.0.0-7560.g6f1c8e8f.el9.x86_ 2.8 MB/s | 834 kB 00:00 2023-12-03T11:09:58.067 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:09:58.068 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.0 MB/s | 912 kB 00:00 2023-12-03T11:09:58.068 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:09:58.090 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:09:58.090 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:09:58.171 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:09:58.171 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:09:58.378 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:09:58.512 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-03T11:09:58.581 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/2 2023-12-03T11:09:59.288 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/2 2023-12-03T11:09:59.288 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/2 2023-12-03T11:09:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-03T11:09:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:59.622 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:09:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-03T11:09:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:09:59.622 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:09:59.932 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2023-12-03T11:10:00.542 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:11 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:00.695 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:18.0.0-7560.g6f1c8e8f.el9 ceph-noarch 262 k 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:00.696 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:10:00.697 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:00.697 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 262 k 2023-12-03T11:10:00.697 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.2 M 2023-12-03T11:10:00.697 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:10:01.246 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-volume-18.0.0-7560.g6f1c8e8f.el9.noarch.rp 478 kB/s | 262 kB 00:00 2023-12-03T11:10:01.246 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:10:01.246 INFO:teuthology.orchestra.run.smithi019.stdout:Total 477 kB/s | 262 kB 00:00 2023-12-03T11:10:01.247 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:10:01.256 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:10:01.256 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:10:01.299 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:10:01.299 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:10:01.526 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:10:01.606 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:10:01.623 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:10:01.623 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:10:01.623 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-03T11:10:01.623 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:02.697 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:18.0.0-7560.g6f1c8e8f.el9.noarch 1/1 2023-12-03T11:10:02.697 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:02.697 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:10:02.697 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:18.0.0-7560.g6f1c8e8f.el9.noarch 2023-12-03T11:10:02.697 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:02.697 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:02.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2023-12-03T11:10:03.587 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:14 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:03.739 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 127 k 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2023-12-03T11:10:03.740 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 456 k 2023-12-03T11:10:03.741 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:10:04.218 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-18.0.0-7560.g6f1c8e8f.el9.x86_64 265 kB/s | 127 kB 00:00 2023-12-03T11:10:04.218 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:10:04.219 INFO:teuthology.orchestra.run.smithi019.stdout:Total 264 kB/s | 127 kB 00:00 2023-12-03T11:10:04.219 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:10:04.225 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:10:04.225 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:10:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:10:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:10:04.432 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:10:04.536 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:04.999 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:05.344 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:05.344 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:05.344 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:10:05.344 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:10:05.344 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:05.345 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:05.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2023-12-03T11:10:06.199 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:17 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:06.300 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:06.354 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:06.356 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:06.356 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:06.425 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2023-12-03T11:10:07.039 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:18 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:07.198 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 31 k 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 k 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 139 k 2023-12-03T11:10:07.199 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:10:07.596 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-18.0.0-7560.g6f1c8e8f.el9.x86_6 78 kB/s | 31 kB 00:00 2023-12-03T11:10:07.597 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:10:07.597 INFO:teuthology.orchestra.run.smithi019.stdout:Total 78 kB/s | 31 kB 00:00 2023-12-03T11:10:07.597 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:10:07.601 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:10:07.601 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:10:07.631 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:10:07.632 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:10:07.728 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:10:07.872 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:08.284 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:08.567 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:08.567 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:08.567 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:10:08.568 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:10:08.568 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:08.568 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:08.773 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2023-12-03T11:10:09.386 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:20 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:09.484 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:09.538 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:09.540 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:09.540 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:09.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2023-12-03T11:10:10.222 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:21 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:10.322 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:10.376 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:10.377 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:10.377 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:10.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2023-12-03T11:10:11.046 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:22 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:11.145 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:11.199 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:11.200 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:11.200 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:11.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2023-12-03T11:10:11.874 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:22 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:11.973 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:12.027 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:12.029 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:12.029 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:12.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2023-12-03T11:10:12.706 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:23 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:12.804 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:12.858 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:12.859 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:12.859 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:12.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2023-12-03T11:10:13.539 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:24 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:13.639 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 is already installed. 2023-12-03T11:10:13.692 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:13.694 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-12-03T11:10:13.694 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:13.762 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2023-12-03T11:10:14.372 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:25 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 85 k 2023-12-03T11:10:14.525 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 85 k 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 256 k 2023-12-03T11:10:14.526 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:10:14.967 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-18.0.0-7560.g6f1c8e8f.el9.x86_64.rpm 194 kB/s | 85 kB 00:00 2023-12-03T11:10:14.967 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:10:14.967 INFO:teuthology.orchestra.run.smithi019.stdout:Total 194 kB/s | 85 kB 00:00 2023-12-03T11:10:14.968 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:10:14.975 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:10:14.976 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:10:15.035 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:10:15.036 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:10:15.184 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:10:15.292 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:15.721 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:15.979 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:15.979 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:15.979 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:10:15.979 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:10:15.979 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:15.979 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:16.210 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2023-12-03T11:10:16.818 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:27 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:16.974 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 3.0 M 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:10:16.975 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:16.976 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:10:16.976 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:16.976 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.0 M 2023-12-03T11:10:16.976 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 13 M 2023-12-03T11:10:16.976 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:10:17.486 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-18.0.0-7560.g6f1c8e8f.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2023-12-03T11:10:17.487 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:10:17.487 INFO:teuthology.orchestra.run.smithi019.stdout:Total 5.8 MB/s | 3.0 MB 00:00 2023-12-03T11:10:17.487 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:10:17.497 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:10:17.497 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:10:17.573 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:10:17.574 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:10:18.017 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:10:18.095 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:18.120 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:18.120 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:10:18.120 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-03T11:10:18.120 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:10:18.120 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:10:18.120 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:19.304 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:19.304 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:19.304 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:10:19.304 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:10:19.304 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:19.304 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:19.617 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2023-12-03T11:10:20.233 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:31 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:10:20.389 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 169 k 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-12-03T11:10:20.390 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:20.391 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 169 k 2023-12-03T11:10:20.391 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 536 k 2023-12-03T11:10:20.391 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:10:20.948 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-18.0.0-7560.g6f1c8e8f.el9.x86_64.rpm 304 kB/s | 169 kB 00:00 2023-12-03T11:10:20.948 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:10:20.948 INFO:teuthology.orchestra.run.smithi019.stdout:Total 303 kB/s | 169 kB 00:00 2023-12-03T11:10:20.949 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:10:20.957 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:10:20.957 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:10:21.018 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:10:21.019 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:10:21.168 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:10:21.272 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:21.723 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:22.039 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/1 2023-12-03T11:10:22.040 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:22.040 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:10:22.040 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:10:22.040 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:22.040 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:10:22.300 DEBUG:teuthology.parallel:result is None 2023-12-03T11:10:22.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f1c8e8f510596dfa1fb82232bce9b991071a48d 2023-12-03T11:10:22.437 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-03T11:10:22.458 INFO:teuthology.orchestra.run.smithi019.stdout:18.0.0-7560.g6f1c8e8f.el9 2023-12-03T11:10:22.459 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7560.g6f1c8e8f.el9 2023-12-03T11:10:22.459 INFO:teuthology.task.install:The correct ceph version 18.0.0-7560.g6f1c8e8f is installed. 2023-12-03T11:10:22.459 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-03T11:10:22.459 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:22.459 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-03T11:10:22.530 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-03T11:10:22.530 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:22.530 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2023-12-03T11:10:22.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-03T11:10:22.671 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-03T11:10:22.671 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:22.671 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-03T11:10:22.742 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-03T11:10:22.813 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-03T11:10:22.813 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:22.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2023-12-03T11:10:22.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-03T11:10:22.953 INFO:teuthology.run_tasks:Running task ceph... 2023-12-03T11:10:23.016 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-03T11:10:23.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2023-12-03T11:10:23.047 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-03T11:10:23.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-03T11:10:23.117 INFO:tasks.ceph:Creating extra log directories... 2023-12-03T11:10:23.117 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-03T11:10:23.194 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-03T11:10:23.194 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-03T11:10:23.194 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096', 'branch': 'wip-matanb-crimson-snaptrimevent-lifetime-v3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': None, 'first_in_suite': False, 'job_id': '7476096', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-snaptrimevent-lifetime-v3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '67b75549b33f045a83f5bf55b677d8842cb7bac4'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9570, 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_67b75549b33f045a83f5bf55b677d8842cb7bac4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '67b75549b33f045a83f5bf55b677d8842cb7bac4', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNlcorkTQv+ZPzLMQe2htM82ypmH64T70NGyY4IZbj7rBEl+FREQYPcmF0GsnyUk/yCA1smrnANdYuh1Yr5mqZ8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '6f1c8e8f510596dfa1fb82232bce9b991071a48d', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-03_10:46:10', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-03T11:10:23.195 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-03T11:10:23.243 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2023-12-03T11:10:23.313 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:23.313 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2023-12-03T11:10:23.368 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-03T11:10:23.369 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-12-03 11:10:18.348556944 +0000 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-12-03 11:02:57.365139873 +0000 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-12-03 11:02:57.365139873 +0000 2023-12-03T11:10:23.424 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-12-03 11:02:57.365139873 +0000 2023-12-03T11:10:23.424 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-03T11:10:23.493 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-12-03T11:10:23.493 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-12-03T11:10:23.493 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000184296 s, 2.8 MB/s 2023-12-03T11:10:23.494 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-03T11:10:23.549 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-12-03 11:10:18.349556934 +0000 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-12-03 11:02:57.883135255 +0000 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-12-03 11:02:57.883135255 +0000 2023-12-03T11:10:23.605 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-12-03 11:02:57.883135255 +0000 2023-12-03T11:10:23.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-03T11:10:23.673 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-12-03T11:10:23.673 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-12-03T11:10:23.673 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000187065 s, 2.7 MB/s 2023-12-03T11:10:23.674 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-03T11:10:23.730 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2023-12-03T11:10:23.785 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-12-03 11:10:18.349556934 +0000 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-12-03 11:02:58.399130654 +0000 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-12-03 11:02:58.399130654 +0000 2023-12-03T11:10:23.786 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-12-03 11:02:58.399130654 +0000 2023-12-03T11:10:23.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-03T11:10:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-12-03T11:10:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-12-03T11:10:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000215886 s, 2.4 MB/s 2023-12-03T11:10:23.856 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-03T11:10:23.911 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2023-12-03T11:10:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-03T11:10:23.967 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:10:23.967 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-03T11:10:23.967 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:10:23.967 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:10:23.967 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-12-03 11:10:18.349556934 +0000 2023-12-03T11:10:23.968 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-12-03 11:02:58.918126027 +0000 2023-12-03T11:10:23.968 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-12-03 11:02:58.918126027 +0000 2023-12-03T11:10:23.968 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-12-03 11:02:58.918126027 +0000 2023-12-03T11:10:23.968 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-03T11:10:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-12-03T11:10:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-12-03T11:10:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000192733 s, 2.7 MB/s 2023-12-03T11:10:24.038 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-03T11:10:24.093 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-03T11:10:24.093 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-03T11:10:24.093 INFO:tasks.ceph:Generating config... 2023-12-03T11:10:24.094 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-03T11:10:24.094 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] err to stderr = True 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] log flush on exit = True 2023-12-03T11:10:24.095 INFO:tasks.ceph:[osd] log to file = False 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] log to stderr = True 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-03T11:10:24.096 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] auth client required = none 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] auth cluster required = none 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] auth service required = none 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] auth supported = none 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] debug asok = 0/0 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] debug auth = 0/0 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-12-03T11:10:24.096 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug client = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug context = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug crush = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug filer = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-12-03T11:10:24.097 INFO:tasks.ceph:[global] debug journal = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mds = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-12-03T11:10:24.098 INFO:tasks.ceph:[global] debug mon = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug monc = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug ms = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug osd = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug rados = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug refs = 0/0 2023-12-03T11:10:24.099 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] debug striper = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] debug timer = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] debug tp = 0/0 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] ms client mode = crc 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-12-03T11:10:24.100 INFO:tasks.ceph:[global] ms service mode = crc 2023-12-03T11:10:24.100 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-03T11:10:24.100 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-03T11:10:24.100 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-12-03T11:10:24.101 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-12-03T11:10:24.101 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-12-03T11:10:24.101 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-12-03T11:10:24.101 INFO:tasks.ceph:Setting up mon.a... 2023-12-03T11:10:24.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-03T11:10:24.188 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2023-12-03T11:10:24.191 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-03T11:10:24.244 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:10:24.274 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.19')] 2023-12-03T11:10:24.275 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-12-03T11:10:24.275 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:24.276 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-03T11:10:24.330 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:24.415 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:24.413+0000 7f142723ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:24.415 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:24.413+0000 7f142723ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:24.415 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:24.413+0000 7f142723ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 99b4367f-52d4-45da-b51a-960d5bce3eec 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = pacific 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 99b4367f-52d4-45da-b51a-960d5bce3eec 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2023-12-03T11:10:24.415117+0000 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:created 2023-12-03T11:10:24.415117+0000 2023-12-03T11:10:24.416 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 16 (pacific) 2023-12-03T11:10:24.417 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2023-12-03T11:10:24.417 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2023-12-03T11:10:24.417 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-03T11:10:24.418 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-03T11:10:24.432 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 99b4367f-52d4-45da-b51a-960d5bce3eec... 2023-12-03T11:10:24.433 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = none 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2023-12-03T11:10:24.533 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2023-12-03T11:10:24.534 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.535 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.19 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: auth client required = none 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: auth cluster required = none 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: auth service required = none 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug asok = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug bdev = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug buffer = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug civetweb = 0/0 2023-12-03T11:10:24.536 INFO:teuthology.orchestra.run.smithi019.stdout: debug client = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug compressor = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug context = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug crush = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug crypto = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug dpdk = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug eventtrace = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug filer = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug filestore = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug finisher = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug fuse = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug heartbeatmap = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug javaclient = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug journal = 0/0 2023-12-03T11:10:24.537 INFO:teuthology.orchestra.run.smithi019.stdout: debug journaler = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug kstore = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug leveldb = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug lockdep = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds balancer = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds locker = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds log = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds log expire = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds migrator = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug memdb = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgrc = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 0/0 2023-12-03T11:10:24.538 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug objclass = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug objectcacher = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug objecter = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug optracker = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug perfcounter = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug rados = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd mirror = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd replay = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug refs = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw = 0/0 2023-12-03T11:10:24.539 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw sync = 0/0 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 0/0 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: debug striper = 0/0 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: debug throttle = 0/0 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: debug timer = 0/0 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: debug tp = 0/0 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: ms client mode = crc 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: ms cluster mode = crc 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon client mode = crc 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon cluster mode = crc 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon service mode = crc 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: ms service mode = crc 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 99b4367f-52d4-45da-b51a-960d5bce3eec 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.540 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2023-12-03T11:10:24.541 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2023-12-03T11:10:24.542 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression algorithm = snappy 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression mode = aggressive 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore rocksdb cf = False 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: crimson seastar smp = 3 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 20 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 20 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug filestore = 0/0 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug journal = 0/0 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 0/0 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 10 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: err to stderr = True 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: log flush on exit = True 2023-12-03T11:10:24.543 INFO:teuthology.orchestra.run.smithi019.stdout: log to file = False 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: log to stderr = True 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: osd shutdown pgref assert = True 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:10:24.544 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 0/0 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 0/0 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2023-12-03T11:10:24.545 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default crimson = True 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2023-12-03T11:10:24.546 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2023-12-03T11:10:24.547 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-03T11:10:24.547 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-03T11:10:24.547 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2023-12-03T11:10:24.550 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-03T11:10:24.550 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-03T11:10:24.655 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-03T11:10:24.656 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:24.656 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-03T11:10:24.671 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:24.671 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-03T11:10:24.726 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:10:24.726 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:24.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-03T11:10:24.726 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:10:24.812 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:24.812 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:24.868 INFO:tasks.ceph:Setting up mon nodes... 2023-12-03T11:10:24.868 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-03T11:10:24.868 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-03T11:10:24.975 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-03T11:10:24.978 INFO:tasks.ceph:Setting up mds nodes... 2023-12-03T11:10:24.978 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-03T11:10:24.978 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-03T11:10:25.030 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-03T11:10:25.051 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-03T11:10:25.051 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-03T11:10:25.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-03T11:10:25.082 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-03T11:10:25.082 INFO:tasks.ceph:role: osd.0 2023-12-03T11:10:25.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:10:25.082 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:10:25.161 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:10:25.674 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2023-12-03T11:10:25.677 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2023-12-03T11:10:25.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-03T11:10:25.802 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-03T11:10:25.841 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-03T11:10:25.912 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-03T11:10:25.912 INFO:tasks.ceph:role: osd.1 2023-12-03T11:10:25.912 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:10:25.912 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-03T11:10:25.990 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:10:25.990 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:10:25.990 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:10:25.991 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:10:26.437 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2023-12-03T11:10:26.439 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2023-12-03T11:10:26.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-03T11:10:26.536 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-03T11:10:26.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-03T11:10:26.641 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-03T11:10:26.641 INFO:tasks.ceph:role: osd.2 2023-12-03T11:10:26.641 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:10:26.641 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-03T11:10:26.719 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:10:26.720 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:10:27.218 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2023-12-03T11:10:27.220 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2023-12-03T11:10:27.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-03T11:10:27.359 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-03T11:10:27.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:27.951 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-03T11:10:27.952 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:27,951 seastar - Reactor backend: linux-aio 2023-12-03T11:10:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:27,970 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-03T11:10:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:27,970 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-03T11:10:27.971 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:27,970 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-03T11:10:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:27,970 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-03T11:10:27.974 INFO:teuthology.orchestra.run.smithi019.stderr:==80827==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-03T11:10:27.980 INFO:teuthology.orchestra.run.smithi019.stderr:==80827==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff058ae000; bottom 0x7f59f26e9000; size: 0x00a5131c5000 (708990226432) 2023-12-03T11:10:27.980 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-12-03T11:10:27.980 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-03T11:10:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0x22f29abc 0x22f2b102 0x22f4a3d7 0x22f4b284 0x22f1b140 0x22f1c33e 0x22f1d429 0x22fff388 0x22e64579 0x22f2e2b4 0x22f2fcff 0x22f30a68 0x22ef88d4 0x22ef7f2c 0x22c08f52 0x24c43510 0x24c4acc6 0x24c4d614 0x24c4e7ed 0x227a8d93 0x227f99f2 0x2281d727 0x2c8b7e35 0x2c955c42 0x2cb79ec4 0x2cb7bbe7 0x2c5d13da 0x2c5d32e3 0x1eebeaf4 0x1eec7891 0x1ed377c2 0x3feaf 0x3ff5f 0x1e7e1a44 2023-12-03T11:10:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:28.535 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:28.535 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-12-03T11:10:28.535 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-12-03T11:10:28.535 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-12-03T11:10:28.536 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:28.536 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:10:28.594 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-03T11:10:28.595 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:28,594 seastar - Reactor backend: linux-aio 2023-12-03T11:10:28.613 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:28,612 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-03T11:10:28.613 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:28,612 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-03T11:10:28.613 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:28,612 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-03T11:10:28.620 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:28,620 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-03T11:10:28.623 INFO:teuthology.orchestra.run.smithi019.stderr:==80825==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-03T11:10:28.629 INFO:teuthology.orchestra.run.smithi019.stderr:==80825==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff058ae000; bottom 0x7f59f04e2000; size: 0x00a5153cc000 (709025906688) 2023-12-03T11:10:28.629 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-12-03T11:10:28.629 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-03T11:10:28.655 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0x248ef 0x94481 0x94d46 0x22c3f36f 0x22c3f697 0x22ccbd35 0x22ccef45 0x22d59167 0x22d5e2cd 0x22d6393e 0x24c43fdb 0x24c4acc6 0x24c4d614 0x24c4e7ed 0x227a8d93 0x227f99f2 0x2281d727 0x2c8b7e35 0x2c955c42 0x2cb79ec4 0x2cb7bbe7 0x2c5d13da 0x2c5d32e3 0x1ed39078 0x3feaf 0x3ff5f 0x1e7e1a44 2023-12-03T11:10:28.655 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:28.716 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x264aa06c 0x264ac6b5 0x264ae4c6 0x264b0d2b 0x262dd55f 0x1edf15cc 0x1ed672ed 0x1ed6b12e 0x2cc99919 2023-12-03T11:10:28.716 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9a4758f4 0xffffffff9a475d8d 0xffffffff999508ea 0xffffffff9a46646c 0xffffffff9a6000ea 2023-12-03T11:10:32.687 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:32.687 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-12-03T11:10:32.687 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-12-03T11:10:32.687 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-12-03T11:10:32.687 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:32.687 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:10:32.688 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 99b4367f-52d4-45da-b51a-960d5bce3eec 2023-12-03T11:10:32.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-03T11:10:32.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:33.304 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-03T11:10:33.304 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,304 seastar - Reactor backend: linux-aio 2023-12-03T11:10:33.324 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,323 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-03T11:10:33.324 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,323 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-03T11:10:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,324 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-03T11:10:33.325 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,324 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-03T11:10:33.327 INFO:teuthology.orchestra.run.smithi019.stderr:==81009==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-03T11:10:33.333 INFO:teuthology.orchestra.run.smithi019.stderr:==81009==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc600b0000; bottom 0x7fcb9578a000; size: 0x0030ca926000 (209557020672) 2023-12-03T11:10:33.333 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-12-03T11:10:33.333 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-03T11:10:33.359 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0x22f1ecb5 0x22f1ffa6 0x22efe4d9 0x22efeb90 0x22f13f3a 0x22f16889 0x22f18838 0x22f3d58c 0x22f5c801 0x22f5d089 0x22f0ac49 0x22f1a269 0x22f1b965 0x22f1c33e 0x22f1d429 0x22fff388 0x22e64579 0x22f2e2b4 0x22f2fcff 0x22f30a68 0x22ef88d4 0x22ef7f2c 0x22c08f52 0x24c43510 0x24c4acc6 0x24c4d614 0x24c4e7ed 0x227a8d93 0x227f99f2 0x2281d727 0x2c8b7e35 0x2c955c42 0x2cb79ec4 0x2cb7bbe7 0x2c5d13da 0x2c5d32e3 0x1eebeaf4 0x1eec7891 0x1ed377c2 0x3feaf 0x3ff5f 0x1e7e1a44 2023-12-03T11:10:33.359 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:33.884 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:33.884 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-12-03T11:10:33.884 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-12-03T11:10:33.884 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-12-03T11:10:33.884 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:33.884 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:10:33.941 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-03T11:10:33.942 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,941 seastar - Reactor backend: linux-aio 2023-12-03T11:10:33.960 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,959 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-03T11:10:33.960 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,959 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-03T11:10:33.960 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,959 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-03T11:10:33.967 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:33,967 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-03T11:10:33.970 INFO:teuthology.orchestra.run.smithi019.stderr:==81007==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-03T11:10:33.976 INFO:teuthology.orchestra.run.smithi019.stderr:==81007==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc600b0000; bottom 0x7fcb93583000; size: 0x0030ccb2d000 (209592700928) 2023-12-03T11:10:33.976 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-12-03T11:10:33.976 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-03T11:10:34.002 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0x22c3be6e 0x22d63c39 0x24c43fdb 0x24c4acc6 0x24c4d614 0x24c4e7ed 0x227a8d93 0x227f99f2 0x2281d727 0x2c8b7e35 0x2c955c42 0x2cb79ec4 0x2cb7bbe7 0x2c5d13da 0x2c5d32e3 0x1ed39078 0x3feaf 0x3ff5f 0x1e7e1a44 2023-12-03T11:10:34.002 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:38.007 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:38.008 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-12-03T11:10:38.008 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-12-03T11:10:38.008 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-12-03T11:10:38.008 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:38.008 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:10:38.008 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 99b4367f-52d4-45da-b51a-960d5bce3eec 2023-12-03T11:10:38.047 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-03T11:10:38.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:38.634 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-03T11:10:38.635 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:38,634 seastar - Reactor backend: linux-aio 2023-12-03T11:10:38.650 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:38,649 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-03T11:10:38.650 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:38,649 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-03T11:10:38.650 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:38,649 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-03T11:10:38.651 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:38,650 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-03T11:10:38.653 INFO:teuthology.orchestra.run.smithi019.stderr:==81191==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-03T11:10:38.659 INFO:teuthology.orchestra.run.smithi019.stderr:==81191==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe36d89000; bottom 0x7fe67cfa8000; size: 0x0017b9de1000 (101902585856) 2023-12-03T11:10:38.659 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-12-03T11:10:38.659 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-03T11:10:38.685 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0x22f1df36 0x22f1ee89 0x22f1ffa6 0x22efe4d9 0x22efeb90 0x22f0ed2b 0x22f10eba 0x22f611a4 0x22f25535 0x22f28b5e 0x22f3cb5f 0x22f5c801 0x22f5d089 0x22f0ac49 0x22f1a269 0x22f1b965 0x22f1c33e 0x22f1d429 0x22fff388 0x22e64579 0x22f2e2b4 0x22f2fcff 0x22f30a68 0x22ef88d4 0x22ef7f2c 0x22c08f52 0x24c43510 0x24c4acc6 0x24c4d614 0x24c4e7ed 0x227a8d93 0x227f99f2 0x2281d727 0x2c8b7e35 0x2c955c42 0x2cb79ec4 0x2cb7bbe7 0x2c5d13da 0x2c5d32e3 0x1eebeaf4 0x1eec7891 0x1ed377c2 0x3feaf 0x3ff5f 0x1e7e1a44 2023-12-03T11:10:38.685 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:39.193 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:39.194 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-12-03T11:10:39.194 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-12-03T11:10:39.194 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-12-03T11:10:39.194 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:39.194 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:10:39.249 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-03T11:10:39.250 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:39,249 seastar - Reactor backend: linux-aio 2023-12-03T11:10:39.268 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:39,267 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-03T11:10:39.268 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:39,267 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-03T11:10:39.268 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:39,267 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-03T11:10:39.275 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-12-03 11:10:39,275 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-03T11:10:39.278 INFO:teuthology.orchestra.run.smithi019.stderr:==81189==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-03T11:10:39.284 INFO:teuthology.orchestra.run.smithi019.stderr:==81189==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe36d89000; bottom 0x7fe67ada1000; size: 0x0017bbfe8000 (101938266112) 2023-12-03T11:10:39.284 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-12-03T11:10:39.284 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-03T11:10:39.310 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0x1ecf76bf 0x1ecf7f38 0x1ec722f3 0x22d5e302 0x22d6393e 0x24c43fdb 0x24c4acc6 0x24c4d614 0x24c4e7ed 0x227a8d93 0x227f99f2 0x2281d727 0x2c8b7e35 0x2c955c42 0x2cb79ec4 0x2cb7bbe7 0x2c5d13da 0x2c5d32e3 0x1ed39078 0x3feaf 0x3ff5f 0x1e7e1a44 2023-12-03T11:10:39.310 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:39.369 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c915b1a 0x2c916ec8 0x2c918416 0x2c91879a 0x2c9188e4 0x2c918d3a 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x264aa06c 0x264ac6b5 0x264ae4c6 0x264b0d2b 0x262dd55f 0x1edf15cc 0x1ed672ed 0x1ed6b12e 0x2cc99919 2023-12-03T11:10:39.370 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-12-03T11:10:43.328 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:43.328 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-12-03T11:10:43.329 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-12-03T11:10:43.329 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-12-03T11:10:43.329 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-12-03T11:10:43.329 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:10:43.329 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 99b4367f-52d4-45da-b51a-960d5bce3eec 2023-12-03T11:10:43.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-03T11:10:43.398 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-03T11:10:43.398 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:43.398 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-03T11:10:43.468 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:43.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-03T11:10:43.538 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:43.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-03T11:10:43.608 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:43.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-03T11:10:43.678 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:43.678 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-03T11:10:43.733 INFO:tasks.ceph:Adding keys to all mons... 2023-12-03T11:10:43.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-03T11:10:43.801 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2023-12-03T11:10:43.801 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQAgYmxlClkfOhAAL1w5WJwOIohgWVX2xJF1yQ== 2023-12-03T11:10:43.801 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2023-12-03T11:10:43.801 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQAkYmxlWWV2KBAAjFDXIiHrSlKSk4VsTIx5aA== 2023-12-03T11:10:43.801 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2023-12-03T11:10:43.801 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQAqYmxlapp4ARAA2FQUUrAjmASkiTXIo6rUAw== 2023-12-03T11:10:43.802 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2023-12-03T11:10:43.802 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQAvYmxlMFLbExAAstEbC6l9WaD0MpUU1hKtSg== 2023-12-03T11:10:43.802 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2023-12-03T11:10:43.802 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQAhYmxl4lPCARAAc1koxTz9pejGInqDI486zw== 2023-12-03T11:10:43.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:10:43.904 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:10:43.966 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:10:44.028 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:10:44.089 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:10:44.153 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-03T11:10:44.154 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-03T11:10:44.184 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:10:44.290 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:44.288+0000 7f51ed433980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.290 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:44.288+0000 7f51ed433980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.308 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-03T11:10:44.338 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:10:44.393 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-03T11:10:44.394 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-03T11:10:44.394 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-03T11:10:44.436 INFO:tasks.ceph.mon.a:Started 2023-12-03T11:10:44.436 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-03T11:10:44.436 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-03T11:10:44.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-03T11:10:44.438 INFO:tasks.ceph.mgr.x:Started 2023-12-03T11:10:44.438 INFO:tasks.ceph:Setting crush tunables to default 2023-12-03T11:10:44.438 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2023-12-03T11:10:44.523 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:44.521+0000 7ff3a962ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.524 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-12-03T11:10:44.521+0000 7fe21e105980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.524 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-12-03T11:10:44.521+0000 7fe21e105980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.524 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-12-03T11:10:44.522+0000 7fe21e105980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.552 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:44.545+0000 7f9ea1915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.586 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:44.585+0000 7f9ea1915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.657 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2023-12-03T11:10:44.668 INFO:tasks.ceph:check_enable_crimson: True 2023-12-03T11:10:44.668 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-12-03T11:10:44.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-12-03T11:10:44.726 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:44.724+0000 7ff3a962ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.726 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:44.724+0000 7ff3a962ef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.778 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:44.772+0000 7fd537824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.780 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:44.778+0000 7fd537824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:44.863 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-12-03T11:10:44.861+0000 7fe215ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-12-03T11:10:44.863 INFO:tasks.ceph.mon.a.smithi019.stderr:Please be aware that this feature is experimental, untested, 2023-12-03T11:10:44.863 INFO:tasks.ceph.mon.a.smithi019.stderr:unsupported, and may result in data corruption, data loss, 2023-12-03T11:10:44.863 INFO:tasks.ceph.mon.a.smithi019.stderr:and/or irreparable damage to your cluster. Do not use 2023-12-03T11:10:44.864 INFO:tasks.ceph.mon.a.smithi019.stderr:feature with important data. 2023-12-03T11:10:44.864 INFO:tasks.ceph.mon.a.smithi019.stderr: 2023-12-03T11:10:45.199 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:45.197+0000 7ff3a962ef00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-03T11:10:45.252 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:45.251+0000 7ff3a962ef00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-03T11:10:45.670 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-03T11:10:45.670 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:45.670 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-03T11:10:45.700 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:45.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-03T11:10:45.772 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:10:45.772 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-03T11:10:45.817 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:45.815+0000 7ff3a962ef00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-03T11:10:45.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new d2f50d5f-3a6b-4ca4-88cb-53f9059c817f 0 2023-12-03T11:10:45.873 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:45.871+0000 7ff3a962ef00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-03T11:10:45.921 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:45.919+0000 7ff3a962ef00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-03T11:10:45.979 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:45.977+0000 7ff3a962ef00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-03T11:10:45.983 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:45.977+0000 7f1432e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:45.985 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:45.984+0000 7f1432e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:46.025 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.023+0000 7ff3a962ef00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-03T11:10:46.066 INFO:teuthology.orchestra.run.smithi019.stdout:0 2023-12-03T11:10:46.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new 6485fd9b-104a-4bd1-a9f3-c50f1e5c9bc3 1 2023-12-03T11:10:46.133 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.131+0000 7ff3a962ef00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-03T11:10:46.187 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:46.181+0000 7f240086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:46.190 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:46.188+0000 7f240086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:46.210 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.208+0000 7ff3a962ef00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-03T11:10:46.258 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.256+0000 7ff3a962ef00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-03T11:10:46.269 INFO:teuthology.orchestra.run.smithi019.stdout:1 2023-12-03T11:10:46.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new e99f2d41-9dcb-4ebb-8d4a-28d8c109edf7 2 2023-12-03T11:10:46.314 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.312+0000 7ff3a962ef00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-03T11:10:46.365 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.364+0000 7ff3a962ef00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-03T11:10:46.391 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:46.384+0000 7f0e5226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:46.393 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:46.391+0000 7f0e5226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:46.473 INFO:teuthology.orchestra.run.smithi019.stdout:2 2023-12-03T11:10:46.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-12-03T11:10:46.559 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-03T11:10:46.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-12-03T11:10:46.602 INFO:tasks.ceph.osd.0:Started 2023-12-03T11:10:46.602 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-12-03T11:10:46.637 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-03T11:10:46.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-12-03T11:10:46.680 INFO:tasks.ceph.osd.1:Started 2023-12-03T11:10:46.680 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-12-03T11:10:46.718 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-03T11:10:46.718 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-12-03T11:10:46.760 INFO:tasks.ceph.osd.2:Started 2023-12-03T11:10:46.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:10:46.794 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:46.792+0000 7ff3a962ef00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-03T11:10:46.973 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:46.965+0000 7f1eba71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:46.977 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:46.975+0000 7f1eba71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:47.020 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.018+0000 7ff3a962ef00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-03T11:10:47.125 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:47.125 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":6,"fsid":"99b4367f-52d4-45da-b51a-960d5bce3eec","created":"2023-12-03T11:10:44.577620+0000","modified":"2023-12-03T11:10:46.471618+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:10:46.471618+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d2f50d5f-3a6b-4ca4-88cb-53f9059c817f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6485fd9b-104a-4bd1-a9f3-c50f1e5c9bc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e99f2d41-9dcb-4ebb-8d4a-28d8c109edf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:10:47.148 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-03T11:10:47.148 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-03T11:10:47.163 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.161+0000 7ff3a962ef00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-03T11:10:47.216 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.214+0000 7ff3a962ef00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-03T11:10:47.326 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.324+0000 7ff3a962ef00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:10:47.429 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.427+0000 7ff3a962ef00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-03T11:10:47.451 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:10:47.552 INFO:teuthology.misc.health.smithi019.stderr:2023-12-03T11:10:47.547+0000 7fa39546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:47.554 INFO:teuthology.misc.health.smithi019.stderr:2023-12-03T11:10:47.552+0000 7fa39546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:47.586 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.584+0000 7ff3a962ef00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:10:47.621 INFO:teuthology.misc.health.smithi019.stdout: 2023-12-03T11:10:47.622 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":6,"fsid":"99b4367f-52d4-45da-b51a-960d5bce3eec","created":"2023-12-03T11:10:44.577620+0000","modified":"2023-12-03T11:10:46.471618+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:10:46.471618+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d2f50d5f-3a6b-4ca4-88cb-53f9059c817f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6485fd9b-104a-4bd1-a9f3-c50f1e5c9bc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e99f2d41-9dcb-4ebb-8d4a-28d8c109edf7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:10:47.632 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-03T11:10:47.635 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.633+0000 7ff3a962ef00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-03T11:10:47.791 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:47.789+0000 7ff3a962ef00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-03T11:10:48.661 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:48.659+0000 7ff3a962ef00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-03T11:10:48.958 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:48.956+0000 7ff3a962ef00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-03T11:10:49.025 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:49.023+0000 7ff3a962ef00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-03T11:10:49.135 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:49.133+0000 7ff3a962ef00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-03T11:10:50.156 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-12-03T11:10:50.154+0000 7ff3497d1640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-03T11:10:53.935 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:10:54.046 INFO:teuthology.misc.health.smithi019.stderr:2023-12-03T11:10:54.041+0000 7f8f4d99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:54.048 INFO:teuthology.misc.health.smithi019.stderr:2023-12-03T11:10:54.046+0000 7f8f4d99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:54.236 INFO:teuthology.misc.health.smithi019.stdout: 2023-12-03T11:10:54.236 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":9,"fsid":"99b4367f-52d4-45da-b51a-960d5bce3eec","created":"2023-12-03T11:10:44.577620+0000","modified":"2023-12-03T11:10:54.166448+0000","last_up_change":"2023-12-03T11:10:52.155694+0000","last_in_change":"2023-12-03T11:10:46.471618+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:10:53.206116+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d2f50d5f-3a6b-4ca4-88cb-53f9059c817f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":580035504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":580035504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":580035504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":580035504}]},"public_addr":"172.21.15.19:6803/580035504","cluster_addr":"172.21.15.19:6802/580035504","heartbeat_back_addr":"172.21.15.19:6813/580035504","heartbeat_front_addr":"172.21.15.19:6812/580035504","state":["exists","up"]},{"osd":1,"uuid":"6485fd9b-104a-4bd1-a9f3-c50f1e5c9bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3059659103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":3059659103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":3059659103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3059659103}]},"public_addr":"172.21.15.19:6804/3059659103","cluster_addr":"172.21.15.19:6805/3059659103","heartbeat_back_addr":"172.21.15.19:6809/3059659103","heartbeat_front_addr":"172.21.15.19:6808/3059659103","state":["exists","up"]},{"osd":2,"uuid":"e99f2d41-9dcb-4ebb-8d4a-28d8c109edf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":77539421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":77539421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":77539421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":77539421}]},"public_addr":"172.21.15.19:6807/77539421","cluster_addr":"172.21.15.19:6806/77539421","heartbeat_back_addr":"172.21.15.19:6811/77539421","heartbeat_front_addr":"172.21.15.19:6810/77539421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:10:54.247 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-03T11:10:54.247 INFO:tasks.ceph:Creating RBD pool 2023-12-03T11:10:54.247 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-03T11:10:54.358 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:54.350+0000 7fdc1239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:54.359 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:54.358+0000 7fdc1239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:55.172 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'rbd' created 2023-12-03T11:10:55.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-03T11:10:55.322 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:55.311+0000 7efcfed1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:55.322+0000 7efcfed1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-03T11:10:56.195 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-03T11:10:56.195 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-03T11:10:56.195 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-03T11:10:56.196 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-03T11:10:56.196 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-03T11:10:56.321 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:56.314+0000 7f542d40f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:56.323 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:56.321+0000 7f542d40f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:56.566 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:56.577 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":603752114},{"type":"v1","addr":"172.21.15.19:6801","nonce":603752114}]},"active_addr":"172.21.15.19:6801/603752114","active_change":"2023-12-03T11:10:49.147868+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":3286742015}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":3529466585}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":3202542048}]}]} 2023-12-03T11:10:56.580 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-03T11:10:56.580 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-03T11:10:56.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:10:56.692 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:56.687+0000 7f1dfe6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:56.693 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:56.692+0000 7f1dfe6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:56.878 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:56.879 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"fsid":"99b4367f-52d4-45da-b51a-960d5bce3eec","created":"2023-12-03T11:10:44.577620+0000","modified":"2023-12-03T11:10:56.177079+0000","last_up_change":"2023-12-03T11:10:52.155694+0000","last_in_change":"2023-12-03T11:10:46.471618+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:10:53.206116+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-03T11:10:54.569126+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d2f50d5f-3a6b-4ca4-88cb-53f9059c817f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":580035504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":580035504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":580035504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":580035504}]},"public_addr":"172.21.15.19:6803/580035504","cluster_addr":"172.21.15.19:6802/580035504","heartbeat_back_addr":"172.21.15.19:6813/580035504","heartbeat_front_addr":"172.21.15.19:6812/580035504","state":["exists","up"]},{"osd":1,"uuid":"6485fd9b-104a-4bd1-a9f3-c50f1e5c9bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3059659103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":3059659103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":3059659103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3059659103}]},"public_addr":"172.21.15.19:6804/3059659103","cluster_addr":"172.21.15.19:6805/3059659103","heartbeat_back_addr":"172.21.15.19:6809/3059659103","heartbeat_front_addr":"172.21.15.19:6808/3059659103","state":["exists","up"]},{"osd":2,"uuid":"e99f2d41-9dcb-4ebb-8d4a-28d8c109edf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":77539421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":77539421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":77539421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":77539421}]},"public_addr":"172.21.15.19:6807/77539421","cluster_addr":"172.21.15.19:6806/77539421","heartbeat_back_addr":"172.21.15.19:6811/77539421","heartbeat_front_addr":"172.21.15.19:6810/77539421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:10:56.891 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-03T11:10:56.891 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:10:57.002 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:56.997+0000 7f82ac59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.004 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.002+0000 7f82ac59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:10:57.191 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"fsid":"99b4367f-52d4-45da-b51a-960d5bce3eec","created":"2023-12-03T11:10:44.577620+0000","modified":"2023-12-03T11:10:57.179991+0000","last_up_change":"2023-12-03T11:10:52.155694+0000","last_in_change":"2023-12-03T11:10:46.471618+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:10:53.206116+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-03T11:10:54.569126+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d2f50d5f-3a6b-4ca4-88cb-53f9059c817f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":580035504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":580035504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":580035504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":580035504}]},"public_addr":"172.21.15.19:6803/580035504","cluster_addr":"172.21.15.19:6802/580035504","heartbeat_back_addr":"172.21.15.19:6813/580035504","heartbeat_front_addr":"172.21.15.19:6812/580035504","state":["exists","up"]},{"osd":1,"uuid":"6485fd9b-104a-4bd1-a9f3-c50f1e5c9bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3059659103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":3059659103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":3059659103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":3059659103}]},"public_addr":"172.21.15.19:6804/3059659103","cluster_addr":"172.21.15.19:6805/3059659103","heartbeat_back_addr":"172.21.15.19:6809/3059659103","heartbeat_front_addr":"172.21.15.19:6808/3059659103","state":["exists","up"]},{"osd":2,"uuid":"e99f2d41-9dcb-4ebb-8d4a-28d8c109edf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":77539421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":77539421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":77539421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":77539421}]},"public_addr":"172.21.15.19:6807/77539421","cluster_addr":"172.21.15.19:6806/77539421","heartbeat_back_addr":"172.21.15.19:6811/77539421","heartbeat_front_addr":"172.21.15.19:6810/77539421","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:10:57.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-03T11:10:57.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-03T11:10:57.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-03T11:10:57.324 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.318+0000 7fcf08d00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.324 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.318+0000 7f75dad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.324 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.318+0000 7fafd966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.326 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.325+0000 7fcf08d00640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.327 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.325+0000 7fafd966d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.327 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.325+0000 7f75dad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.355 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-12-03T11:10:57.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:10:57.356 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-12-03T11:10:57.356 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:10:57.359 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-12-03T11:10:57.359 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-03T11:10:57.473 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.466+0000 7f7b09715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.475 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.474+0000 7f7b09715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.516 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.510+0000 7efd20d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.516 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.510+0000 7fe5ce02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.518 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.517+0000 7efd20d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.519 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:57.517+0000 7fe5ce02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:57.666 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-12-03T11:10:57.683 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-12-03T11:10:57.711 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-12-03T11:10:57.712 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-12-03T11:10:57.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-12-03T11:10:57.725 DEBUG:teuthology.parallel:result is None 2023-12-03T11:10:57.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-12-03T11:10:58.684 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:10:58.727 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:10:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:58.806+0000 7f726646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:58.815+0000 7f726646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:58.851 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:58.845+0000 7f3c8151e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:58.853 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:10:58.851+0000 7f3c8151e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:10:59.003 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-12-03T11:10:59.016 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-12-03T11:10:59.038 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-12-03T11:10:59.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-12-03T11:11:00.016 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:11:00.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:11:00.148 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.140+0000 7f642826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.150 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.148+0000 7f642826d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.185 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.179+0000 7f140302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.187 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.185+0000 7f140302d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.349 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-12-03T11:11:00.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-12-03T11:11:00.362 DEBUG:teuthology.parallel:result is None 2023-12-03T11:11:00.378 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-12-03T11:11:00.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-12-03T11:11:00.391 DEBUG:teuthology.parallel:result is None 2023-12-03T11:11:00.391 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-03T11:11:00.391 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T11:11:00.543 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.539+0000 7fd72a31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.547 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.545+0000 7fd72a31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.727 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:00.727 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2023-12-03T11:11:00.739 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-03T11:10:59.152262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59914,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.948683"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.308659+0000","last_change":"2023-12-03T11:10:56.308706+0000","last_active":"2023-12-03T11:10:56.308659+0000","last_peered":"2023-12-03T11:10:56.308659+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.306637+0000","last_became_peered":"2023-12-03T11:10:56.306637+0000","last_unstale":"2023-12-03T11:10:56.308659+0000","last_undegraded":"2023-12-03T11:10:56.308659+0000","last_fullsized":"2023-12-03T11:10:56.308659+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.306207+0000","last_change":"2023-12-03T11:10:56.306263+0000","last_active":"2023-12-03T11:10:56.306207+0000","last_peered":"2023-12-03T11:10:56.306207+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.304360+0000","last_became_peered":"2023-12-03T11:10:56.304360+0000","last_unstale":"2023-12-03T11:10:56.306207+0000","last_undegraded":"2023-12-03T11:10:56.306207+0000","last_fullsized":"2023-12-03T11:10:56.306207+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.298022+0000","last_change":"2023-12-03T11:10:56.298098+0000","last_active":"2023-12-03T11:10:56.298022+0000","last_peered":"2023-12-03T11:10:56.298022+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.287806+0000","last_became_peered":"2023-12-03T11:10:56.287806+0000","last_unstale":"2023-12-03T11:10:56.298022+0000","last_undegraded":"2023-12-03T11:10:56.298022+0000","last_fullsized":"2023-12-03T11:10:56.298022+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.323086+0000","last_change":"2023-12-03T11:10:56.323185+0000","last_active":"2023-12-03T11:10:56.323086+0000","last_peered":"2023-12-03T11:10:56.323086+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.319437+0000","last_became_peered":"2023-12-03T11:10:56.319437+0000","last_unstale":"2023-12-03T11:10:56.323086+0000","last_undegraded":"2023-12-03T11:10:56.323086+0000","last_fullsized":"2023-12-03T11:10:56.323086+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.289551+0000","last_change":"2023-12-03T11:10:56.289601+0000","last_active":"2023-12-03T11:10:56.289551+0000","last_peered":"2023-12-03T11:10:56.289551+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.287040+0000","last_became_peered":"2023-12-03T11:10:56.287040+0000","last_unstale":"2023-12-03T11:10:56.289551+0000","last_undegraded":"2023-12-03T11:10:56.289551+0000","last_fullsized":"2023-12-03T11:10:56.289551+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-03T11:10:57.188190+0000","last_change":"2023-12-03T11:10:57.188217+0000","last_active":"2023-12-03T11:10:57.188190+0000","last_peered":"2023-12-03T11:10:57.188190+0000","last_clean":"2023-12-03T11:10:57.188190+0000","last_became_active":"2023-12-03T11:10:56.256523+0000","last_became_peered":"2023-12-03T11:10:56.256523+0000","last_unstale":"2023-12-03T11:10:57.188190+0000","last_undegraded":"2023-12-03T11:10:57.188190+0000","last_fullsized":"2023-12-03T11:10:57.188190+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-03T11:10:57.187662+0000","last_change":"2023-12-03T11:10:57.187692+0000","last_active":"2023-12-03T11:10:57.187662+0000","last_peered":"2023-12-03T11:10:57.187662+0000","last_clean":"2023-12-03T11:10:57.187662+0000","last_became_active":"2023-12-03T11:10:56.249980+0000","last_became_peered":"2023-12-03T11:10:56.249980+0000","last_unstale":"2023-12-03T11:10:57.187662+0000","last_undegraded":"2023-12-03T11:10:57.187662+0000","last_fullsized":"2023-12-03T11:10:57.187662+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.245506+0000","last_change":"2023-12-03T11:10:56.245558+0000","last_active":"2023-12-03T11:10:56.245506+0000","last_peered":"2023-12-03T11:10:56.245506+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.243732+0000","last_became_peered":"2023-12-03T11:10:56.243732+0000","last_unstale":"2023-12-03T11:10:56.245506+0000","last_undegraded":"2023-12-03T11:10:56.245506+0000","last_fullsized":"2023-12-03T11:10:56.245506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.339266+0000","last_change":"2023-12-03T11:10:56.208389+0000","last_active":"2023-12-03T11:10:56.339266+0000","last_peered":"2023-12-03T11:10:56.339266+0000","last_clean":"2023-12-03T11:10:56.339266+0000","last_became_active":"2023-12-03T11:10:55.230381+0000","last_became_peered":"2023-12-03T11:10:55.230381+0000","last_unstale":"2023-12-03T11:10:56.339266+0000","last_undegraded":"2023-12-03T11:10:56.339266+0000","last_fullsized":"2023-12-03T11:10:56.339266+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:54.166448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:54.166448+0000","last_clean_scrub_stamp":"2023-12-03T11:10:54.166448+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-03T11:11:00.739 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T11:11:00.852 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.849+0000 7fc5f902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:00.854 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:00.852+0000 7fc5f902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:01.038 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:01.039 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2023-12-03T11:11:01.050 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-03T11:10:59.152262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59914,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.948683"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.308659+0000","last_change":"2023-12-03T11:10:56.308706+0000","last_active":"2023-12-03T11:10:56.308659+0000","last_peered":"2023-12-03T11:10:56.308659+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.306637+0000","last_became_peered":"2023-12-03T11:10:56.306637+0000","last_unstale":"2023-12-03T11:10:56.308659+0000","last_undegraded":"2023-12-03T11:10:56.308659+0000","last_fullsized":"2023-12-03T11:10:56.308659+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.306207+0000","last_change":"2023-12-03T11:10:56.306263+0000","last_active":"2023-12-03T11:10:56.306207+0000","last_peered":"2023-12-03T11:10:56.306207+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.304360+0000","last_became_peered":"2023-12-03T11:10:56.304360+0000","last_unstale":"2023-12-03T11:10:56.306207+0000","last_undegraded":"2023-12-03T11:10:56.306207+0000","last_fullsized":"2023-12-03T11:10:56.306207+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.298022+0000","last_change":"2023-12-03T11:10:56.298098+0000","last_active":"2023-12-03T11:10:56.298022+0000","last_peered":"2023-12-03T11:10:56.298022+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.287806+0000","last_became_peered":"2023-12-03T11:10:56.287806+0000","last_unstale":"2023-12-03T11:10:56.298022+0000","last_undegraded":"2023-12-03T11:10:56.298022+0000","last_fullsized":"2023-12-03T11:10:56.298022+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.323086+0000","last_change":"2023-12-03T11:10:56.323185+0000","last_active":"2023-12-03T11:10:56.323086+0000","last_peered":"2023-12-03T11:10:56.323086+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.319437+0000","last_became_peered":"2023-12-03T11:10:56.319437+0000","last_unstale":"2023-12-03T11:10:56.323086+0000","last_undegraded":"2023-12-03T11:10:56.323086+0000","last_fullsized":"2023-12-03T11:10:56.323086+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.289551+0000","last_change":"2023-12-03T11:10:56.289601+0000","last_active":"2023-12-03T11:10:56.289551+0000","last_peered":"2023-12-03T11:10:56.289551+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.287040+0000","last_became_peered":"2023-12-03T11:10:56.287040+0000","last_unstale":"2023-12-03T11:10:56.289551+0000","last_undegraded":"2023-12-03T11:10:56.289551+0000","last_fullsized":"2023-12-03T11:10:56.289551+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-03T11:10:57.188190+0000","last_change":"2023-12-03T11:10:57.188217+0000","last_active":"2023-12-03T11:10:57.188190+0000","last_peered":"2023-12-03T11:10:57.188190+0000","last_clean":"2023-12-03T11:10:57.188190+0000","last_became_active":"2023-12-03T11:10:56.256523+0000","last_became_peered":"2023-12-03T11:10:56.256523+0000","last_unstale":"2023-12-03T11:10:57.188190+0000","last_undegraded":"2023-12-03T11:10:57.188190+0000","last_fullsized":"2023-12-03T11:10:57.188190+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-03T11:10:57.187662+0000","last_change":"2023-12-03T11:10:57.187692+0000","last_active":"2023-12-03T11:10:57.187662+0000","last_peered":"2023-12-03T11:10:57.187662+0000","last_clean":"2023-12-03T11:10:57.187662+0000","last_became_active":"2023-12-03T11:10:56.249980+0000","last_became_peered":"2023-12-03T11:10:56.249980+0000","last_unstale":"2023-12-03T11:10:57.187662+0000","last_undegraded":"2023-12-03T11:10:57.187662+0000","last_fullsized":"2023-12-03T11:10:57.187662+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.245506+0000","last_change":"2023-12-03T11:10:56.245558+0000","last_active":"2023-12-03T11:10:56.245506+0000","last_peered":"2023-12-03T11:10:56.245506+0000","last_clean":"2023-12-03T11:10:55.170432+0000","last_became_active":"2023-12-03T11:10:56.243732+0000","last_became_peered":"2023-12-03T11:10:56.243732+0000","last_unstale":"2023-12-03T11:10:56.245506+0000","last_undegraded":"2023-12-03T11:10:56.245506+0000","last_fullsized":"2023-12-03T11:10:56.245506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:55.170432+0000","last_clean_scrub_stamp":"2023-12-03T11:10:55.170432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-03T11:10:56.339266+0000","last_change":"2023-12-03T11:10:56.208389+0000","last_active":"2023-12-03T11:10:56.339266+0000","last_peered":"2023-12-03T11:10:56.339266+0000","last_clean":"2023-12-03T11:10:56.339266+0000","last_became_active":"2023-12-03T11:10:55.230381+0000","last_became_peered":"2023-12-03T11:10:55.230381+0000","last_unstale":"2023-12-03T11:10:56.339266+0000","last_undegraded":"2023-12-03T11:10:56.339266+0000","last_fullsized":"2023-12-03T11:10:56.339266+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:10:54.166448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:10:54.166448+0000","last_clean_scrub_stamp":"2023-12-03T11:10:54.166448+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-03T11:11:01.051 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-03T11:11:01.051 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-03T11:11:01.051 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-03T11:11:01.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-03T11:11:01.162 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:01.159+0000 7f32fa86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:01.164 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:11:01.163+0000 7f32fa86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:11:01.385 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:01.385 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-03T11:11:01.398 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-03T11:11:01.398 INFO:teuthology.run_tasks:Running task cbt... 2023-12-03T11:11:01.411 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi019.front.sepia.ceph.com', 'osds': ['ubuntu@smithi019.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi019.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi019.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi019.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-12-03T11:11:01.412 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-12-03T11:11:01.429 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:11:01.429 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-03T11:11:01.482 DEBUG:teuthology.orchestra.run.smithi019:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-12-03T11:11:01.536 INFO:teuthology.orchestra.run.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-12-03T11:11:02.073 INFO:tasks.cbt:Installing collectl 2023-12-03T11:11:02.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-12-03T11:11:02.218 INFO:teuthology.orchestra.run.smithi019.stderr:--2023-12-03 11:11:02-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-12-03T11:11:02.360 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2023-12-03T11:11:02.382 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-12-03T11:11:02.634 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-03T11:11:02.635 INFO:teuthology.orchestra.run.smithi019.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlbGJG40DJYXaJ2wNR9cYHRZI8mml1_qAVfL0rO-mNdMASkYpevB5y4athZqVLcyMXpzzEY6ZQ9tyv49XztW4juZ4bmA%3D%3D&use_mirror=cytranet&r= [following] 2023-12-03T11:11:02.635 INFO:teuthology.orchestra.run.smithi019.stderr:--2023-12-03 11:11:02-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlbGJG40DJYXaJ2wNR9cYHRZI8mml1_qAVfL0rO-mNdMASkYpevB5y4athZqVLcyMXpzzEY6ZQ9tyv49XztW4juZ4bmA%3D%3D&use_mirror=cytranet&r= 2023-12-03T11:11:02.635 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-12-03T11:11:02.712 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-12-03T11:11:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-03T11:11:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-12-03T11:11:02.971 INFO:teuthology.orchestra.run.smithi019.stderr:--2023-12-03 11:11:02-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-12-03T11:11:02.998 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2023-12-03T11:11:03.069 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2023-12-03T11:11:03.393 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 200 OK 2023-12-03T11:11:03.394 INFO:teuthology.orchestra.run.smithi019.stderr:Length: 620693 (606K) [application/x-gzip] 2023-12-03T11:11:03.394 INFO:teuthology.orchestra.run.smithi019.stderr:Saving to: ‘collectl.tar.gz’ 2023-12-03T11:11:03.394 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:11:03.464 INFO:teuthology.orchestra.run.smithi019.stderr: 0K .......... .......... .......... .......... .......... 8% 710K 1s 2023-12-03T11:11:03.533 INFO:teuthology.orchestra.run.smithi019.stderr: 50K .......... .......... .......... .......... .......... 16% 723K 1s 2023-12-03T11:11:03.535 INFO:teuthology.orchestra.run.smithi019.stderr: 100K .......... .......... .......... .......... .......... 24% 19.1M 0s 2023-12-03T11:11:03.603 INFO:teuthology.orchestra.run.smithi019.stderr: 150K .......... .......... .......... .......... .......... 32% 750K 0s 2023-12-03T11:11:03.604 INFO:teuthology.orchestra.run.smithi019.stderr: 200K .......... .......... .......... .......... .......... 41% 45.7M 0s 2023-12-03T11:11:03.604 INFO:teuthology.orchestra.run.smithi019.stderr: 250K .......... .......... .......... .......... .......... 49% 65.0M 0s 2023-12-03T11:11:03.605 INFO:teuthology.orchestra.run.smithi019.stderr: 300K .......... .......... .......... .......... .......... 57% 29.6M 0s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 350K .......... .......... .......... .......... .......... 65% 759K 0s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 400K .......... .......... .......... .......... .......... 74% 79.0M 0s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 450K .......... .......... .......... .......... .......... 82% 92.0M 0s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 500K .......... .......... .......... .......... .......... 90% 81.0M 0s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 550K .......... .......... .......... .......... .......... 98% 93.1M 0s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 600K ...... 100% 11.4T=0.3s 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03 11:11:03 (2.11 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-12-03T11:11:03.676 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:11:03.717 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/ 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/collectl.conf 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/SlowProc.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/WhySummary.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Socket.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/colmux.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Tutorial.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Gexpr.html 2023-12-03T11:11:03.718 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Import.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/style.css 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/FileNaming.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/TheMath.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Environmental.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Process.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/HiResTime.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/slub.jpg 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Lustre.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Documentation.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Inodes.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Graphite.html 2023-12-03T11:11:03.719 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/InputFiles.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Export.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Interrupts.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Startup.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Performance.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Messages.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Logging.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-detail.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/CPUs.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Matrix.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-brief.html 2023-12-03T11:11:03.720 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Misc.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Examples.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OpenStack.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Exceptions.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Playback.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Hello.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Sockets.html 2023-12-03T11:11:03.721 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Network.html 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Infiniband.html 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Memory.html 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Disks.html 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/service/ 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/service/collectl.service 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/ 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-suse 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-debian 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-generic 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/misc.ph 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/gexpr.ph 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/statsd.ph 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/envrules.std 2023-12-03T11:11:03.722 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/hello.ph 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vmsum.ph 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/proctree.ph 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/README-WINDOWS 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/formatit.ph 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/colmux 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/INSTALL 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/graphite.ph 2023-12-03T11:11:03.723 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/RELEASE-collectl 2023-12-03T11:11:03.724 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vmstat.ph 2023-12-03T11:11:03.724 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/README 2023-12-03T11:11:03.724 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/ 2023-12-03T11:11:03.724 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/collectl.1 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/colmux.1 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/lexpr.ph 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/client.pl 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/collectl 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/COPYING 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vnet.ph 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/GPL 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/UNINSTALL 2023-12-03T11:11:03.725 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/ARTISTIC 2023-12-03T11:11:03.782 INFO:teuthology.orchestra.run.smithi019.stderr:./INSTALL: line 90: chkconfig: command not found 2023-12-03T11:11:03.783 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-12-03T11:11:04.391 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:07:15 ago on Sun 03 Dec 2023 11:03:49 AM UTC. 2023-12-03T11:11:04.491 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-12-03T11:11:04.494 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-12-03T11:11:04.554 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout: librbd-devel x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 23 k 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout: perf x86_64 5.14.0-388.el9 appstream 7.6 M 2023-12-03T11:11:04.555 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout: libradospp-devel x86_64 2:18.0.0-7560.g6f1c8e8f.el9 ceph 30 k 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout:Install 5 Packages 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 7.7 M 2023-12-03T11:11:04.556 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 11 M 2023-12-03T11:11:04.557 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-12-03T11:11:05.167 INFO:teuthology.orchestra.run.smithi019.stdout:(1/5): libradospp-devel-18.0.0-7560.g6f1c8e8f.e 137 kB/s | 30 kB 00:00 2023-12-03T11:11:05.276 INFO:teuthology.orchestra.run.smithi019.stdout:(2/5): pdsh-2.34-7.el9.x86_64.rpm 1.1 MB/s | 118 kB 00:00 2023-12-03T11:11:05.292 INFO:teuthology.orchestra.run.smithi019.stdout:(3/5): librbd-devel-18.0.0-7560.g6f1c8e8f.el9.x 67 kB/s | 23 kB 00:00 2023-12-03T11:11:05.318 INFO:teuthology.orchestra.run.smithi019.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 315 kB/s | 13 kB 00:00 2023-12-03T11:11:06.485 INFO:teuthology.orchestra.run.smithi019.stdout:(5/5): perf-5.14.0-388.el9.x86_64.rpm 4.9 MB/s | 7.6 MB 00:01 2023-12-03T11:11:06.487 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:11:06.488 INFO:teuthology.orchestra.run.smithi019.stdout:Total 4.0 MB/s | 7.7 MB 00:01 2023-12-03T11:11:06.572 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-12-03T11:11:06.597 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-12-03T11:11:06.597 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-12-03T11:11:06.806 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-12-03T11:11:06.806 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-12-03T11:11:07.203 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-12-03T11:11:07.287 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-12-03T11:11:07.317 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-12-03T11:11:07.349 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradospp-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 3/5 2023-12-03T11:11:07.709 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librbd-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 4/5 2023-12-03T11:11:07.831 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : perf-5.14.0-388.el9.x86_64 5/5 2023-12-03T11:11:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: perf-5.14.0-388.el9.x86_64 5/5 2023-12-03T11:11:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradospp-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 1/5 2023-12-03T11:11:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2/5 2023-12-03T11:11:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : perf-5.14.0-388.el9.x86_64 3/5 2023-12-03T11:11:08.825 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: libradospp-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: librbd-devel-2:18.0.0-7560.g6f1c8e8f.el9.x86_64 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-2.34-7.el9.x86_64 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: perf-5.14.0-388.el9.x86_64 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-12-03T11:11:09.156 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-12-03T11:11:09.493 INFO:tasks.cbt:benchmark: radosbench 2023-12-03T11:11:09.493 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-12-03T11:11:09.525 DEBUG:teuthology.orchestra.run.smithi019:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-03T11:11:09.874 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - Settings.cluster: 2023-12-03T11:11:09.874 INFO:teuthology.orchestra.run.smithi019.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-12-03T11:11:09.874 INFO:teuthology.orchestra.run.smithi019.stderr: 'clients': ['ubuntu@smithi019.front.sepia.ceph.com'], 2023-12-03T11:11:09.874 INFO:teuthology.orchestra.run.smithi019.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'head': 'ubuntu@smithi019.front.sepia.ceph.com', 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'is_teuthology': True, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'iterations': 1, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'mons': ['ubuntu@smithi019.front.sepia.ceph.com'], 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'osds': ['ubuntu@smithi019.front.sepia.ceph.com'], 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'osds_per_node': 3, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'pid_dir': '/var/run/ceph', 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'pgp_size': 256, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'replication': 'replicated'}}, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'rebuild_every_test': False, 2023-12-03T11:11:09.875 INFO:teuthology.orchestra.run.smithi019.stderr: 'rgws': {'ubuntu@smithi019.front.sepia.ceph.com': None}, 2023-12-03T11:11:09.876 INFO:teuthology.orchestra.run.smithi019.stderr: 'tmp_dir': '/tmp/cbt', 2023-12-03T11:11:09.876 INFO:teuthology.orchestra.run.smithi019.stderr: 'use_existing': True, 2023-12-03T11:11:09.876 INFO:teuthology.orchestra.run.smithi019.stderr: 'user': 'ubuntu'} 2023-12-03T11:11:09.895 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-12-03T11:11:09.896 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:11:09.897 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:11:09.897 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2023-12-03T11:11:09.900 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:11:09.901 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-03T11:11:09.902 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:11:09.902 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:11:09.902 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2023-12-03T11:11:09.905 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:11:09.905 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-03T11:11:09.905 INFO:teuthology.orchestra.run.smithi019.stderr:11:11:09 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-03T11:12:09.965 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:09.966 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-03T11:12:09.999 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:09.999 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-03T11:12:10.045 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.045 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-12-03T11:12:10.047 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.047 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-12-03T11:12:10.049 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.049 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.049 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2023-12-03T11:12:10.070 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.070 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi019.front.sepia.ceph.com ; done 2023-12-03T11:12:10.074 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.074 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.074 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-12-03T11:12:10.099 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - WARNING - cbt - sudo killall -9 rados 2023-12-03T11:12:10.099 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - WARNING - cbt - error 1 seen, continuing anyway... 2023-12-03T11:12:10.119 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2023-12-03T11:12:10.119 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-12-03T11:12:10.119 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.119 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.120 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-12-03T11:12:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2023-12-03T11:12:10.142 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.142 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-03T11:12:10.143 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.143 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.143 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2023-12-03T11:12:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-03T11:12:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-03T11:12:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-12-03T11:12:10.451 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.451 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:10.451 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-12-03T11:12:11.632 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:11 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:11.632 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:11 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:11.632 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:11 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-12-03T11:12:12.639 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:12 - INFO - cbt - Checking Health after pool creation. 2023-12-03T11:12:17.641 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:17 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:17.641 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:17 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:17.641 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-03T11:12:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:17 - INFO - cbt - Final Pool Health Check. 2023-12-03T11:12:22.998 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:22 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:22.998 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:22 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:22.998 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-03T11:12:23.317 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.317 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-03T11:12:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.336 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-03T11:12:23.371 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.371 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-12-03T11:12:23.373 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.373 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-12-03T11:12:23.375 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.375 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.375 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-12-03T11:12:23.501 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:23.501 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-12-03T11:12:24.104 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:24.105 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:24.105 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-12-03T11:12:24.398 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:24.398 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:24.398 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2023-12-03T11:12:24.458 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:24.458 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:24.458 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2023-12-03T11:12:25.209 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.209 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.209 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2023-12-03T11:12:25.211 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.211 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-03T11:12:25.213 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.213 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.213 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2023-12-03T11:12:25.216 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-03T11:12:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-03T11:12:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - INFO - cbt - Running radosbench write test. 2023-12-03T11:12:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.217 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2023-12-03T11:12:25.218 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.218 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:12:25.218 INFO:teuthology.orchestra.run.smithi019.stderr:11:12:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2023-12-03T11:13:26.353 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.354 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-03T11:13:26.386 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.386 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-03T11:13:26.426 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.426 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2023-12-03T11:13:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.428 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2023-12-03T11:13:26.429 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.430 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.430 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2023-12-03T11:13:26.749 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.765 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.765 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2023-12-03T11:13:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi019.front.sepia.ceph.com ; done 2023-12-03T11:13:26.815 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - INFO - cbt - Convert results to json format. 2023-12-03T11:13:26.815 INFO:teuthology.orchestra.run.smithi019.stderr:11:13:26 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:26.836 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-12-03T11:13:26.845 DEBUG:teuthology.orchestra.run.smithi019:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-12-03T11:13:27.035 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-12-03T11:13:27.035 INFO:tasks.cbt_performance:reading cbt preferences 2023-12-03T11:13:27.035 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-12-03T11:13:27.089 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi019.front.sepia.ceph.com 2023-12-03T11:13:27.089 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi019.front.sepia.ceph.com 2023-12-03T11:13:27.090 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:13:27.090 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi019.front.sepia.ceph.com of=/dev/stdout 2023-12-03T11:13:27.106 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-03T11:13:27.107 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-12-03T11:13:27.161 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-03T11:13:27.162 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-03T11:13:27.162 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-12-03T11:13:27.162 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi019.front.sepia.ceph.com of=/dev/stdout 2023-12-03T11:13:27.216 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-03T11:13:27.217 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-12-03T11:13:27.272 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-03T11:13:27.272 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-03T11:13:27.281 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-03T11:13:27.287 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-03T11:13:27.288 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-03T11:13:27.297 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-03T11:13:27.424 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:13:27.416+0000 7febb6f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:13:27.426 INFO:teuthology.orchestra.run.smithi019.stderr:2023-12-03T11:13:27.424+0000 7febb6f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-03T11:13:27.629 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-03T11:13:27.630 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-03T11:13:27.630 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-03T11:13:27.630 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T11:13:30.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-03T11:13:33.731 INFO:tasks.ceph.osd.0:Stopped 2023-12-03T11:13:33.731 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-03T11:13:33.731 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T11:13:35.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-03T11:13:35.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-03T11:13:39.833 INFO:tasks.ceph.osd.1:Stopped 2023-12-03T11:13:39.834 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-03T11:13:39.834 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T11:13:41.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-03T11:13:41.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-03T11:13:45.935 INFO:tasks.ceph.osd.2:Stopped 2023-12-03T11:13:45.936 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-03T11:13:45.936 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-03T11:13:45.936 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T11:13:45.967 INFO:tasks.ceph.mgr.x:Stopped 2023-12-03T11:13:45.967 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-03T11:13:45.968 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-03T11:13:45.968 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T11:13:46.021 INFO:tasks.ceph.mon.a:Stopped 2023-12-03T11:13:46.021 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-03T11:13:46.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-12-03T11:13:46.110 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:46.110 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-03T11:13:46.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-03T11:13:46.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:46.536 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-03T11:13:46.623 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:46.623 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-03T11:13:46.680 INFO:tasks.ceph:Archiving mon data... 2023-12-03T11:13:46.681 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096/data/mon.a.tgz 2023-12-03T11:13:46.681 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2023-12-03T11:13:46.703 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.n2fOwBrL7q 2023-12-03T11:13:46.703 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.n2fOwBrL7q 2023-12-03T11:13:46.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.n2fOwBrL7q 2023-12-03T11:13:46.935 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.n2fOwBrL7q is 176KB 2023-12-03T11:13:46.988 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.n2fOwBrL7q 2023-12-03T11:13:47.003 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-03T11:13:47.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-03T11:13:47.155 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-03T11:13:47.156 INFO:tasks.ceph:Archiving crash dumps... 2023-12-03T11:13:47.157 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096/remote/smithi019/crash 2023-12-03T11:13:47.158 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-03T11:13:47.225 INFO:tasks.ceph:Compressing logs... 2023-12-03T11:13:47.225 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-03T11:13:47.306 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81990.log 2023-12-03T11:13:47.306 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-03T11:13:47.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.81990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83332.log 2023-12-03T11:13:47.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82909.log 2023-12-03T11:13:47.307 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.83332.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80341.log 2023-12-03T11:13:47.307 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82786.log 2023-12-03T11:13:47.307 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81950.log 2023-12-03T11:13:47.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83122.log 2023-12-03T11:13:47.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82910.log 2023-12-03T11:13:47.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84236.log 2023-12-03T11:13:47.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84301.log 2023-12-03T11:13:47.308 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-03T11:13:47.309 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84725.log 2023-12-03T11:13:47.309 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82996.log 2023-12-03T11:13:47.309 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82683.log 2023-12-03T11:13:47.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83036.log 2023-12-03T11:13:47.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84188.log 2023-12-03T11:13:47.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81910.log 2023-12-03T11:13:47.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83226.log 2023-12-03T11:13:47.310 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83206.log 2023-12-03T11:13:47.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-03T11:13:47.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83142.log 2023-12-03T11:13:47.311 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84300.log 2023-12-03T11:13:47.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82585.log 2023-12-03T11:13:47.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84151.log 2023-12-03T11:13:47.312 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82908.log 2023-12-03T11:13:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82148.log 2023-12-03T11:13:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81800.log 2023-12-03T11:13:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-03T11:13:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82178.log 2023-12-03T11:13:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83290.log 2023-12-03T11:13:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83374.log 2023-12-03T11:13:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83035.log 2023-12-03T11:13:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82641.log 2023-12-03T11:13:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-03T11:13:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82744.log 2023-12-03T11:13:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-03T11:13:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-03T11:13:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82828.log 2023-12-03T11:13:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81712.log 2023-12-03T11:13:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84212.log 2023-12-03T11:13:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84086.log 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.80341.log: /var/log/ceph/ceph-client.admin.82909.log: /var/log/ceph/ceph-client.admin.84301.log: /var/log/ceph/ceph-client.admin.84212.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.84725.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.84300.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.0.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84301.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84725.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.2.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84300.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81990.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83332.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.83122.log: /var/log/ceph/ceph-client.admin.82786.log: /var/log/ceph/ceph-client.admin.82996.log: /var/log/ceph/ceph-client.admin.82910.log: /var/log/ceph/ceph-client.admin.81950.log: /var/log/ceph/ceph-client.admin.84236.log: /var/log/ceph/ceph-client.admin.83036.log: /var/log/ceph/ceph-client.admin.81910.log: /var/log/ceph/ceph-client.admin.82683.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82909.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.83142.log: /var/log/ceph/ceph-client.admin.83226.log: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-03T11:13:47.328 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.84188.log: /var/log/ceph/ceph-client.admin.81800.log: /var/log/ceph/ceph-client.admin.82585.log: /var/log/ceph/ceph-client.admin.84151.log: /var/log/ceph/ceph-client.admin.82148.log: /var/log/ceph/ceph-client.admin.82178.log: /var/log/ceph/ceph-client.admin.82641.log: /var/log/ceph/ceph-client.admin.83374.log: /var/log/ceph/ceph-client.admin.83206.log: /var/log/ceph/ceph-client.admin.82744.log: /var/log/ceph/ceph-client.admin.83035.log: /var/log/ceph/ceph-client.admin.83290.log: /var/log/ceph/ceph-client.admin.81712.log: /var/log/ceph/ceph-client.admin.82828.log: /var/log/ceph/ceph-client.admin.82908.log: /var/log/ceph/ceph-client.admin.84086.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83122.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82786.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82996.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82910.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83036.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82683.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83142.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83226.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82585.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82641.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83374.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83206.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82744.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83290.log.gz 2023-12-03T11:13:47.329 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83035.log.gz 2023-12-03T11:13:47.330 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81712.log.gz 2023-12-03T11:13:47.330 INFO:teuthology.orchestra.run.smithi019.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.82828.log.gz 2023-12-03T11:13:47.330 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82908.log.gz 2023-12-03T11:13:47.332 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.84086.log.gz 2023-12-03T11:13:47.332 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84151.log.gz 2023-12-03T11:13:47.332 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84188.log.gz 2023-12-03T11:13:47.332 INFO:teuthology.orchestra.run.smithi019.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-03T11:13:47.334 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82148.log.gz 2023-12-03T11:13:47.335 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82178.log.gz 2023-12-03T11:13:47.339 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81910.log.gz 2023-12-03T11:13:47.339 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81950.log.gz 2023-12-03T11:13:47.344 INFO:teuthology.orchestra.run.smithi019.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80341.log.gz 2023-12-03T11:13:47.350 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.84212.log.gz 2023-12-03T11:13:47.350 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84236.log.gz 2023-12-03T11:13:47.354 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81800.log.gz 2023-12-03T11:13:47.360 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-03T11:13:47.646 INFO:teuthology.orchestra.run.smithi019.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-03T11:13:48.951 INFO:teuthology.orchestra.run.smithi019.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-03T11:13:49.079 INFO:teuthology.orchestra.run.smithi019.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-03T11:13:49.088 INFO:teuthology.orchestra.run.smithi019.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-03T11:13:49.091 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-12-03T11:13:49.091 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m1.811s 2023-12-03T11:13:49.091 INFO:teuthology.orchestra.run.smithi019.stderr:user 0m3.788s 2023-12-03T11:13:49.091 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m0.190s 2023-12-03T11:13:49.091 INFO:tasks.ceph:Archiving logs... 2023-12-03T11:13:49.091 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096/remote/smithi019/log 2023-12-03T11:13:52.501 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-03T11:13:53.173 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-03T11:13:53.182 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-03T11:13:53.182 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-03T11:13:53.244 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-03T11:13:53.253 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-03T11:13:53.254 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T11:13:53.275 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2023-12-03T11:13:53.347 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T11:13:53.347 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2023-12-03T11:13:53.347 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +1771us[+1771us] +/- 98ms 2023-12-03T11:13:53.347 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -462us[ -460us] +/- 65ms 2023-12-03T11:13:53.347 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -1083us[-1081us] +/- 97ms 2023-12-03T11:13:53.347 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:13:53.356 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-03T11:13:53.365 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-03T11:13:53.366 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-03T11:13:53.384 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-03T11:13:53.597 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T11:13:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:13:53.635 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-03T11:13:53.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-03T11:13:53.668 INFO:teuthology.task.internal:Duration was 988.676710 seconds 2023-12-03T11:13:53.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-03T11:13:53.701 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-03T11:13:53.701 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-03T11:13:53.833 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T11:13:54.292 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-03T11:13:54.292 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2023-12-03T11:13:54.293 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-03T11:13:54.314 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-03T11:13:54.314 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-03T11:13:54.389 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-03T11:13:54.389 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-03T11:13:54.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-03T11:13:54.563 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-03T11:13:54.563 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-03T11:13:54.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-03T11:13:54.620 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-03T11:13:54.666 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2023-12-03T11:13:54.698 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-03T11:13:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:13:54.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-03T11:13:54.723 INFO:teuthology.task.internal:Transferring archived files... 2023-12-03T11:13:54.723 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-12-03_10:46:10-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v3-distro-crimson-smithi/7476096/remote/smithi019 2023-12-03T11:13:54.724 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-03T11:13:54.822 INFO:teuthology.task.internal:Removing archive directory... 2023-12-03T11:13:54.822 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-03T11:13:54.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-03T11:13:54.882 INFO:teuthology.task.internal:Not uploading archives. 2023-12-03T11:13:54.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-03T11:13:54.902 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-03T11:13:54.902 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-03T11:13:54.922 INFO:teuthology.orchestra.run.smithi019.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 11:13 /home/ubuntu/cephtest 2023-12-03T11:13:54.923 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-03T11:13:54.935 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-03T11:13:55.005 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-snaptrimevent-lifetime-v3&var-job_id_selected=7476096 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 988.6767101287842 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-12-03T11:13:55.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T11:13:55.088 INFO:teuthology.run:pass