2023-10-15T15:10:56.300 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-15T15:10:56.301 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} --archive /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419 --owner scheduled_matan@teuthology --verbose --name matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi -- /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419/orig.config.yaml 2023-10-15T15:10:56.344 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-15T15:10:56.424 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} email: null first_in_suite: false job_id: '7428419' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 984ba140e1395377d07322ca248175a686ef49e1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 984ba140e1395377d07322ca248175a686ef49e1 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 9b943423e52ce7834e53a46d2315dd8f44efe018 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4843 sha1: 984ba140e1395377d07322ca248175a686ef49e1 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 9b943423e52ce7834e53a46d2315dd8f44efe018 targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKbU7DwDL1Koa9YrtiOTquFRHJu98yxKRBdS4efslIEShzuWkURIZGQP0dye2Ml8uIylp5u+U5BauRujswOGiI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-15_14:39:28 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-15T15:10:56.425 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa; will attempt to use it 2023-10-15T15:10:56.426 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa/tasks 2023-10-15T15:10:56.426 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-15T15:10:56.428 INFO:teuthology.task.internal:Checking packages... 2023-10-15T15:10:56.450 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '984ba140e1395377d07322ca248175a686ef49e1' 2023-10-15T15:10:56.450 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-15T15:10:56.451 INFO:teuthology.packaging:ref: None 2023-10-15T15:10:56.451 INFO:teuthology.packaging:tag: None 2023-10-15T15:10:56.452 INFO:teuthology.packaging:branch: main 2023-10-15T15:10:56.452 INFO:teuthology.packaging:sha1: 984ba140e1395377d07322ca248175a686ef49e1 2023-10-15T15:10:56.452 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2023-10-15T15:10:56.772 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6729.g984ba140 2023-10-15T15:10:56.772 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-15T15:10:56.779 INFO:teuthology.task.internal:no buildpackages task found 2023-10-15T15:10:56.779 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-15T15:10:56.801 INFO:teuthology.task.internal:Saving configuration 2023-10-15T15:10:56.817 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-15T15:10:56.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-15T15:10:56.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-15 15:06:19.569325', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKbU7DwDL1Koa9YrtiOTquFRHJu98yxKRBdS4efslIEShzuWkURIZGQP0dye2Ml8uIylp5u+U5BauRujswOGiI='} 2023-10-15T15:10:56.849 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-15T15:10:56.855 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-15T15:10:56.856 INFO:teuthology.run_tasks:Running task console_log... 2023-10-15T15:10:56.892 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7a511989d0>, signals=[15]) 2023-10-15T15:10:56.894 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-15T15:10:56.899 INFO:teuthology.task.internal:Opening connections... 2023-10-15T15:10:56.899 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:10:56.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:10:57.002 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-15T15:10:57.009 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2023-10-15T15:10:57.071 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2023-10-15T15:10:57.071 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2023-10-15T15:10:57.128 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2023-10-15T15:10:57.129 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2023-10-15T15:10:57.129 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2023-10-15T15:10:57.129 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2023-10-15T15:10:57.130 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2023-10-15T15:10:57.130 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2023-10-15T15:10:57.130 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-15T15:10:57.130 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2023-10-15T15:10:57.130 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2023-10-15T15:10:57.131 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-15T15:10:57.131 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2023-10-15T15:10:57.131 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-15T15:10:57.131 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-15T15:10:57.131 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-15T15:10:57.132 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2023-10-15T15:10:57.166 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-15T15:10:57.174 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-15T15:10:57.179 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-15T15:10:57.179 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2023-10-15T15:10:57.195 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-15T15:10:57.201 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-15T15:10:57.201 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2023-10-15T15:10:57.265 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-15T15:10:57.296 INFO:teuthology.run_tasks:Running task kernel... 2023-10-15T15:10:57.310 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-15T15:10:57.310 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-15T15:10:57.311 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-15T15:10:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:10:57.328 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2023-10-15T15:10:57.385 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2023-10-15T15:10:57.386 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-361.el9.x86_64 2023-10-15T15:10:57.386 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2023-10-15T15:10:59.559 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-15T15:11:01.050 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2023-10-15T15:11:02.441 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 7.9 MB 00:01 2023-10-15T15:11:10.758 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 18 MB 00:07 2023-10-15T15:11:14.842 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 5.7 MB 00:00 2023-10-15T15:11:16.226 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 14 kB 00:00 2023-10-15T15:11:16.961 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 30 MB/s | 19 MB 00:00 2023-10-15T15:11:21.595 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-10-15T15:11:23.734 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-15T15:11:23.735 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-15T15:11:23.814 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:11:23.821 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:11:23.821 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:11:23.822 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:11:23.822 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:11:23.822 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-375.el9 CentOS-BaseOS 5.3 M 2023-10-15T15:11:23.822 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:11:23.822 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-375.el9 CentOS-BaseOS 20 M 2023-10-15T15:11:23.823 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-375.el9 CentOS-BaseOS 37 M 2023-10-15T15:11:23.823 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-375.el9 CentOS-BaseOS 32 M 2023-10-15T15:11:23.823 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:11:23.823 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:11:23.823 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:11:23.824 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2023-10-15T15:11:23.824 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:11:23.825 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 95 M 2023-10-15T15:11:23.825 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 120 M 2023-10-15T15:11:23.825 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:11:24.196 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-375.el9.x86_64.rpm 14 MB/s | 5.3 MB 00:00 2023-10-15T15:11:24.671 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-core-5.14.0-375.el9.x86_64.rpm 23 MB/s | 20 MB 00:00 2023-10-15T15:11:25.329 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-5.14.0-375.el9.x86_64.rpm 25 MB/s | 37 MB 00:01 2023-10-15T15:11:25.746 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-core-5.14.0-375.el9.x86_6 21 MB/s | 32 MB 00:01 2023-10-15T15:11:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:11:25.750 INFO:teuthology.orchestra.run.smithi019.stdout:Total 49 MB/s | 95 MB 00:01 2023-10-15T15:11:25.750 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:11:26.185 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:11:26.185 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:11:27.205 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:11:27.205 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:11:29.710 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:11:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-375.el9.x86_64 1/4 2023-10-15T15:11:30.506 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-15T15:11:33.392 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-15T15:11:33.582 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-15T15:11:38.513 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-15T15:11:38.608 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:11:43.568 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:12:14.360 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:12:14.405 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:12:14.823 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:12:14.823 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-375.el9.x86_64 1/4 2023-10-15T15:12:14.823 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-15T15:12:14.823 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-15T15:12:15.282 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-15T15:12:15.282 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:12:15.283 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:12:15.283 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-375.el9.x86_64 2023-10-15T15:12:15.284 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-375.el9.x86_64 2023-10-15T15:12:15.284 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-375.el9.x86_64 2023-10-15T15:12:15.284 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-375.el9.x86_64 2023-10-15T15:12:15.285 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:12:15.285 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:12:15.777 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2023-10-15T15:12:15.912 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-15T15:12:15.912 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-15T15:12:15.913 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-375.el9.x86_64 2023-10-15T15:12:15.913 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-15T15:12:15.913 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-15T15:12:15.913 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-15T15:12:15.914 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2023-10-15T15:12:15.914 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2023-10-15T15:12:16.023 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-15T15:12:16.024 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-15T15:12:16.024 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2023-10-15T15:12:16.067 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2023-10-15T15:12:16.067 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2023-10-15T15:12:16.068 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2023-10-15T15:12:16.068 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2023-10-15T15:12:16.068 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2023-10-15T15:12:16.068 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-15T15:12:16.068 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2023-10-15T15:12:16.069 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2023-10-15T15:12:16.069 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2023-10-15T15:12:16.069 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-15T15:12:16.069 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-15T15:12:16.069 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-15T15:12:16.070 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-15T15:12:16.070 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2023-10-15T15:12:16.070 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2023-10-15T15:12:16.070 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-15T15:12:16.070 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2023-10-15T15:12:16.071 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2023-10-15T15:12:16.071 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:12:16.071 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-15T15:12:16.071 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-15T15:12:16.071 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-15T15:12:16.072 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2023-10-15T15:12:16.072 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:12:16.072 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2023-10-15T15:12:16.073 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-15T15:12:16.073 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-375.el9.x86_64 2023-10-15T15:12:16.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-15T15:12:16.754 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2023-10-15T15:12:18.034 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-15T15:12:18.056 INFO:teuthology.orchestra.run.smithi019.stderr:done 2023-10-15T15:12:18.058 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2023-10-15T15:12:18.074 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.HBeSLJqfdN 2023-10-15T15:12:18.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HBeSLJqfdN 2023-10-15T15:12:18.146 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.HBeSLJqfdN 2023-10-15T15:12:18.285 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.HBeSLJqfdN is 6KB 2023-10-15T15:12:18.335 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.HBeSLJqfdN 2023-10-15T15:12:18.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2023-10-15T15:12:18.426 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-15T15:12:18.427 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-15T15:12:18.427 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-15T15:12:18.427 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64.conf 2023-10-15T15:12:18.428 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64 2023-10-15T15:12:18.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2023-10-15T15:12:48.618 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-15T15:12:48.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-10-15T15:12:48.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:13:06.938 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2023-10-15T15:13:15.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-10-15T15:13:15.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:14:16.004 DEBUG:teuthology.orchestra.remote:timed out 2023-10-15T15:14:28.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-10-15T15:14:28.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:14:28.446 DEBUG:teuthology.orchestra.run.smithi019:> true 2023-10-15T15:14:28.829 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-10-15T15:14:28.829 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-375.el9.x86_64"... 2023-10-15T15:14:28.830 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2023-10-15T15:14:28.846 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-375.el9.x86_64 2023-10-15T15:14:28.846 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-375.el9.x86_64 vs 5.14.0-375.el9.x86_64 2023-10-15T15:14:28.847 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-15T15:14:28.847 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-15T15:14:29.848 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-15T15:14:29.849 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-15T15:14:29.903 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2023-10-15T15:14:29.948 DEBUG:teuthology.parallel:result is None 2023-10-15T15:14:29.948 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-15T15:14:29.955 INFO:teuthology.task.internal:Creating test directory... 2023-10-15T15:14:29.955 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-15T15:14:29.972 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-15T15:14:29.978 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-15T15:14:29.989 INFO:teuthology.task.internal:Creating archive directory... 2023-10-15T15:14:29.989 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-15T15:14:30.042 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-15T15:14:30.049 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-15T15:14:30.049 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-15T15:14:30.141 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-15T15:14:30.158 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-15T15:14:30.160 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-15T15:14:30.167 INFO:teuthology.task.internal:Configuring sudo... 2023-10-15T15:14:30.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-15T15:14:30.199 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-15T15:14:30.209 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-15T15:14:30.210 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-15T15:14:30.258 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-15T15:14:30.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-15T15:14:30.396 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-15T15:14:30.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-15T15:14:30.524 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:14:30.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-15T15:14:30.595 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2023-10-15T15:14:30.682 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-15T15:14:30.879 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-15T15:14:30.888 INFO:teuthology.task.internal:Starting timer... 2023-10-15T15:14:30.888 INFO:teuthology.run_tasks:Running task pcp... 2023-10-15T15:14:30.929 INFO:teuthology.run_tasks:Running task selinux... 2023-10-15T15:14:30.938 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2023-10-15T15:14:30.992 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2023-10-15T15:14:30.995 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-15T15:14:30.995 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2023-10-15T15:14:31.024 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2023-10-15T15:14:31.024 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive'} 2023-10-15T15:14:31.025 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-15T15:14:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:14:31.096 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-15T15:14:31.097 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2023-10-15T15:14:31.176 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-15T15:14:31.187 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-15T15:14:31.193 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-15T15:14:31.343 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-15T15:14:31.357 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-15T15:14:31.359 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-15T15:20:01.521 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com')] 2023-10-15T15:20:01.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-10-15T15:20:01.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-15T15:20:01.623 DEBUG:teuthology.orchestra.run.smithi019:> true 2023-10-15T15:20:01.690 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2023-10-15T15:20:01.691 INFO:teuthology.run_tasks:Running task clock... 2023-10-15T15:20:01.704 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-15T15:20:01.704 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-15T15:20:01.705 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-15T15:20:01.766 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-15T15:20:01.787 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-15T15:20:01.824 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2023-10-15T15:20:01.842 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2023-10-15T15:20:01.862 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-15T15:20:01.882 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-15T15:20:01.930 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2023-10-15T15:20:01.933 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-15T15:20:01.934 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2023-10-15T15:20:01.934 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:20:01.934 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:20:01.934 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:20:01.935 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:20:01.936 INFO:teuthology.run_tasks:Running task install... 2023-10-15T15:20:01.944 DEBUG:teuthology.task.install:project ceph 2023-10-15T15:20:01.944 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'}} 2023-10-15T15:20:01.945 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'} 2023-10-15T15:20:01.945 INFO:teuthology.task.install:Using flavor: crimson 2023-10-15T15:20:01.951 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-15T15:20:01.951 INFO:teuthology.task.install:extra packages: [] 2023-10-15T15:20:01.951 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '984ba140e1395377d07322ca248175a686ef49e1', 'tag': None, 'wait_for_package': False} 2023-10-15T15:20:01.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=984ba140e1395377d07322ca248175a686ef49e1 2023-10-15T15:20:02.180 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/ 2023-10-15T15:20:02.181 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6729.g984ba140 2023-10-15T15:20:02.354 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/984ba140e1395377d07322ca248175a686ef49e1/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-15T15:20:02.355 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:20:02.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-15T15:20:02.387 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-15T15:20:02.387 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/984ba140e1395377d07322ca248175a686ef49e1/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-15T15:20:02.459 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-15T15:20:02.546 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-15T15:20:02.576 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2023-10-15T15:20:02.578 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2023-10-15T15:20:02.827 INFO:teuthology.orchestra.run.smithi019.stdout:72 files removed 2023-10-15T15:20:02.853 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2023-10-15T15:20:03.608 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 194 kB/s | 81 kB 00:00 2023-10-15T15:20:03.975 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 30 kB/s | 9.4 kB 00:00 2023-10-15T15:20:04.401 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2023-10-15T15:20:04.878 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 14 MB/s | 6.3 MB 00:00 2023-10-15T15:20:06.320 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2023-10-15T15:20:09.069 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 7.9 MB 00:02 2023-10-15T15:20:11.939 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 18 MB 00:01 2023-10-15T15:20:16.730 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 5.7 MB 00:01 2023-10-15T15:20:18.056 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 14 kB 00:00 2023-10-15T15:20:20.886 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 19 MB 00:02 2023-10-15T15:20:25.549 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2023-10-15T15:20:28.001 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:20:28.004 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:20:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:20:28.005 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:20:28.005 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:20:28.005 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:18.0.0-6729.g984ba140.el9 ceph 8.4 M 2023-10-15T15:20:28.006 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2023-10-15T15:20:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 3.1 M 2023-10-15T15:20:28.006 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 3.0 M 2023-10-15T15:20:28.006 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:20:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options 2023-10-15T15:20:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-15T15:20:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:18.0.0-6729.g984ba140.el9 ceph 6.6 M 2023-10-15T15:20:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:18.0.0-6729.g984ba140.el9 ceph 17 M 2023-10-15T15:20:28.007 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:18.0.0-6729.g984ba140.el9 ceph 24 k 2023-10-15T15:20:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-15T15:20:28.008 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-15T15:20:28.009 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-15T15:20:28.009 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-15T15:20:28.009 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 726 k 2023-10-15T15:20:28.009 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-15T15:20:28.009 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-15T15:20:28.010 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-15T15:20:28.010 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 470 k 2023-10-15T15:20:28.010 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-15T15:20:28.011 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:18.0.0-6729.g984ba140.el9 ceph 4.5 M 2023-10-15T15:20:28.011 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-15T15:20:28.011 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-15T15:20:28.011 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-15T15:20:28.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse 2023-10-15T15:20:28.012 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 2:18.0.0-6729.g984ba140.el9 ceph 45 k 2023-10-15T15:20:28.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:18.0.0-6729.g984ba140.el9 ceph 127 k 2023-10-15T15:20:28.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:18.0.0-6729.g984ba140.el9 ceph 162 k 2023-10-15T15:20:28.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-15T15:20:28.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:18.0.0-6729.g984ba140.el9 ceph 321 k 2023-10-15T15:20:28.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:18.0.0-6729.g984ba140.el9 ceph 297 k 2023-10-15T15:20:28.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:18.0.0-6729.g984ba140.el9 ceph 99 k 2023-10-15T15:20:28.013 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-15T15:20:28.013 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-15T15:20:28.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:28.014 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:20:28.014 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:20:28.014 INFO:teuthology.orchestra.run.smithi019.stdout:Install 28 Packages 2023-10-15T15:20:28.014 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2023-10-15T15:20:28.014 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:28.015 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 54 M 2023-10-15T15:20:28.015 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:20:29.242 INFO:teuthology.orchestra.run.smithi019.stdout:(1/30): ceph-radosgw-18.0.0-6729.g984ba140.el9. 11 MB/s | 8.4 MB 00:00 2023-10-15T15:20:29.292 INFO:teuthology.orchestra.run.smithi019.stdout:(2/30): ceph-selinux-18.0.0-6729.g984ba140.el9. 484 kB/s | 24 kB 00:00 2023-10-15T15:20:29.375 INFO:teuthology.orchestra.run.smithi019.stdout:(3/30): libcephfs2-18.0.0-6729.g984ba140.el9.x8 8.5 MB/s | 726 kB 00:00 2023-10-15T15:20:29.434 INFO:teuthology.orchestra.run.smithi019.stdout:(4/30): libradosstriper1-18.0.0-6729.g984ba140. 7.9 MB/s | 470 kB 00:00 2023-10-15T15:20:29.762 INFO:teuthology.orchestra.run.smithi019.stdout:(5/30): librgw2-18.0.0-6729.g984ba140.el9.x86_6 14 MB/s | 4.5 MB 00:00 2023-10-15T15:20:29.843 INFO:teuthology.orchestra.run.smithi019.stdout:(6/30): ceph-base-18.0.0-6729.g984ba140.el9.x86 4.7 MB/s | 6.6 MB 00:01 2023-10-15T15:20:29.860 INFO:teuthology.orchestra.run.smithi019.stdout:(7/30): python3-ceph-argparse-18.0.0-6729.g984b 450 kB/s | 45 kB 00:00 2023-10-15T15:20:29.911 INFO:teuthology.orchestra.run.smithi019.stdout:(8/30): python3-cephfs-18.0.0-6729.g984ba140.el 3.2 MB/s | 162 kB 00:00 2023-10-15T15:20:29.936 INFO:teuthology.orchestra.run.smithi019.stdout:(9/30): python3-ceph-common-18.0.0-6729.g984ba1 1.3 MB/s | 127 kB 00:00 2023-10-15T15:20:29.969 INFO:teuthology.orchestra.run.smithi019.stdout:(10/30): python3-rados-18.0.0-6729.g984ba140.el 5.4 MB/s | 321 kB 00:00 2023-10-15T15:20:30.011 INFO:teuthology.orchestra.run.smithi019.stdout:(11/30): python3-rgw-18.0.0-6729.g984ba140.el9. 2.3 MB/s | 99 kB 00:00 2023-10-15T15:20:30.045 INFO:teuthology.orchestra.run.smithi019.stdout:(12/30): python3-rbd-18.0.0-6729.g984ba140.el9. 2.7 MB/s | 297 kB 00:00 2023-10-15T15:20:30.212 INFO:teuthology.orchestra.run.smithi019.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2023-10-15T15:20:30.237 INFO:teuthology.orchestra.run.smithi019.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 830 kB/s | 159 kB 00:00 2023-10-15T15:20:30.288 INFO:teuthology.orchestra.run.smithi019.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-10-15T15:20:30.313 INFO:teuthology.orchestra.run.smithi019.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-10-15T15:20:30.330 INFO:teuthology.orchestra.run.smithi019.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 997 kB/s | 41 kB 00:00 2023-10-15T15:20:30.455 INFO:teuthology.orchestra.run.smithi019.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2023-10-15T15:20:30.705 INFO:teuthology.orchestra.run.smithi019.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 295 kB/s | 111 kB 00:00 2023-10-15T15:20:30.914 INFO:teuthology.orchestra.run.smithi019.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 674 kB/s | 309 kB 00:00 2023-10-15T15:20:30.989 INFO:teuthology.orchestra.run.smithi019.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 325 kB/s | 24 kB 00:00 2023-10-15T15:20:31.073 INFO:teuthology.orchestra.run.smithi019.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2023-10-15T15:20:31.148 INFO:teuthology.orchestra.run.smithi019.stdout:(23/30): libarrow-9.0.0-9.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2023-10-15T15:20:31.173 INFO:teuthology.orchestra.run.smithi019.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2023-10-15T15:20:31.349 INFO:teuthology.orchestra.run.smithi019.stdout:(25/30): ceph-common-18.0.0-6729.g984ba140.el9. 6.0 MB/s | 17 MB 00:02 2023-10-15T15:20:31.374 INFO:teuthology.orchestra.run.smithi019.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 957 kB/s | 192 kB 00:00 2023-10-15T15:20:31.416 INFO:teuthology.orchestra.run.smithi019.stdout:(27/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.1 MB/s | 837 kB 00:00 2023-10-15T15:20:31.708 INFO:teuthology.orchestra.run.smithi019.stdout:(28/30): librbd1-18.0.0-6729.g984ba140.el9.x86_ 11 MB/s | 3.0 MB 00:00 2023-10-15T15:20:31.991 INFO:teuthology.orchestra.run.smithi019.stdout:(29/30): librados2-18.0.0-6729.g984ba140.el9.x8 5.0 MB/s | 3.1 MB 00:00 2023-10-15T15:20:32.034 INFO:teuthology.orchestra.run.smithi019.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 2.3 MB/s | 1.6 MB 00:00 2023-10-15T15:20:32.039 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:20:32.039 INFO:teuthology.orchestra.run.smithi019.stdout:Total 13 MB/s | 54 MB 00:04 2023-10-15T15:20:32.051 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:20:32.089 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:20:32.090 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:20:32.591 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:20:32.592 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:20:34.127 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:20:34.554 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-15T15:20:34.602 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:18.0.0-6729.g984ba140.el9.x86_64 2/32 2023-10-15T15:20:34.695 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:18.0.0-6729.g984ba140.el9.x86_64 2/32 2023-10-15T15:20:34.793 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 3/32 2023-10-15T15:20:34.829 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 4/32 2023-10-15T15:20:34.875 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 4/32 2023-10-15T15:20:34.916 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-15T15:20:34.950 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-15T15:20:34.988 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-15T15:20:35.032 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:18.0.0-6729.g984ba140.el9. 8/32 2023-10-15T15:20:35.101 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 9/32 2023-10-15T15:20:35.124 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_6 10/32 2023-10-15T15:20:35.212 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_6 10/32 2023-10-15T15:20:35.248 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-15T15:20:35.366 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-15T15:20:35.399 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-15T15:20:35.440 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-15T15:20:35.472 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-15T15:20:35.499 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-15T15:20:35.535 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-15T15:20:35.568 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-15T15:20:35.896 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-15T15:20:35.931 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 20/32 2023-10-15T15:20:36.022 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 20/32 2023-10-15T15:20:36.052 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 21/32 2023-10-15T15:20:36.088 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-15T15:20:36.545 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-15T15:20:36.578 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 24/32 2023-10-15T15:20:36.646 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 24/32 2023-10-15T15:20:36.725 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 25/32 2023-10-15T15:20:36.761 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:18.0.0-6729.g984ba140.el9.x8 26/32 2023-10-15T15:20:38.888 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 27/32 2023-10-15T15:20:38.941 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 27/32 2023-10-15T15:20:39.707 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 27/32 2023-10-15T15:20:39.747 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 28/32 2023-10-15T15:20:39.941 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 28/32 2023-10-15T15:20:39.941 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-15T15:20:39.942 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-15T15:20:39.942 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:39.973 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 29/32 2023-10-15T15:20:50.041 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 29/32 2023-10-15T15:20:50.041 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2023-10-15T15:20:50.042 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2023-10-15T15:20:50.042 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2023-10-15T15:20:50.042 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2023-10-15T15:20:50.042 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2023-10-15T15:20:50.042 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2023-10-15T15:20:50.043 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2023-10-15T15:20:50.043 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:50.905 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 30/32 2023-10-15T15:20:50.931 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 30/32 2023-10-15T15:20:50.931 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:20:50.932 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-15T15:20:50.932 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-15T15:20:50.932 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-15T15:20:50.932 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:50.933 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-15T15:20:51.023 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-15T15:20:51.024 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-15T15:20:52.172 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-15T15:20:52.173 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 1/32 2023-10-15T15:20:52.173 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 2/32 2023-10-15T15:20:52.173 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 3/32 2023-10-15T15:20:52.173 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 4/32 2023-10-15T15:20:52.174 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 5/32 2023-10-15T15:20:52.174 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_6 6/32 2023-10-15T15:20:52.174 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 7/32 2023-10-15T15:20:52.174 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6729.g984ba140.el9. 8/32 2023-10-15T15:20:52.174 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:18.0.0-6729.g984ba140.el9.x8 9/32 2023-10-15T15:20:52.175 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 10/32 2023-10-15T15:20:52.175 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 11/32 2023-10-15T15:20:52.175 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 12/32 2023-10-15T15:20:52.175 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 13/32 2023-10-15T15:20:52.175 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-15T15:20:52.176 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-15T15:20:52.176 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-15T15:20:52.176 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-15T15:20:52.176 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-15T15:20:52.176 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-15T15:20:52.177 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-15T15:20:52.177 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-15T15:20:52.177 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-15T15:20:52.177 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-15T15:20:52.177 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-15T15:20:52.178 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-15T15:20:52.178 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-15T15:20:52.178 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-15T15:20:52.179 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-15T15:20:52.179 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:18.0.0-6729.g984ba140.el9.x86_64 29/32 2023-10-15T15:20:52.179 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-15T15:20:52.180 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 31/32 2023-10-15T15:20:53.247 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-15T15:20:53.247 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:53.247 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2023-10-15T15:20:53.248 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.248 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.248 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:20:53.248 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-15T15:20:53.248 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.249 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.249 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.249 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.249 INFO:teuthology.orchestra.run.smithi019.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-15T15:20:53.249 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-15T15:20:53.250 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-15T15:20:53.250 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-15T15:20:53.250 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.251 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-15T15:20:53.251 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-15T15:20:53.251 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-15T15:20:53.251 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.251 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-15T15:20:53.251 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.252 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-15T15:20:53.252 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-15T15:20:53.252 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-15T15:20:53.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.253 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.253 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-15T15:20:53.253 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.253 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.254 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:20:53.254 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2023-10-15T15:20:53.254 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-15T15:20:53.255 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:53.255 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:20:53.611 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2023-10-15T15:20:54.219 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:29 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:20:54.371 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:20:54.372 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:20:54.372 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:20:54.372 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:20:54.372 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:20:54.373 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:18.0.0-6729.g984ba140.el9 ceph 117 M 2023-10-15T15:20:54.373 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:20:54.373 INFO:teuthology.orchestra.run.smithi019.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-15T15:20:54.373 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-15T15:20:54.373 INFO:teuthology.orchestra.run.smithi019.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-15T15:20:54.374 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:18.0.0-6729.g984ba140.el9 ceph 161 k 2023-10-15T15:20:54.374 INFO:teuthology.orchestra.run.smithi019.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-15T15:20:54.374 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-15T15:20:54.374 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-15T15:20:54.374 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-15T15:20:54.375 INFO:teuthology.orchestra.run.smithi019.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-15T15:20:54.375 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:54.375 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:20:54.375 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:20:54.376 INFO:teuthology.orchestra.run.smithi019.stdout:Install 10 Packages 2023-10-15T15:20:54.376 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:20:54.376 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 120 M 2023-10-15T15:20:54.377 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.1 G 2023-10-15T15:20:54.377 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:20:54.696 INFO:teuthology.orchestra.run.smithi019.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.2 MB/s | 418 kB 00:00 2023-10-15T15:20:54.738 INFO:teuthology.orchestra.run.smithi019.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-10-15T15:20:54.780 INFO:teuthology.orchestra.run.smithi019.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-10-15T15:20:54.822 INFO:teuthology.orchestra.run.smithi019.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2023-10-15T15:20:54.855 INFO:teuthology.orchestra.run.smithi019.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-15T15:20:54.905 INFO:teuthology.orchestra.run.smithi019.stdout:(6/10): libcephsqlite-18.0.0-6729.g984ba140.el9 407 kB/s | 161 kB 00:00 2023-10-15T15:20:55.355 INFO:teuthology.orchestra.run.smithi019.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 373 kB/s | 187 kB 00:00 2023-10-15T15:20:55.897 INFO:teuthology.orchestra.run.smithi019.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.3 MB/s | 1.3 MB 00:00 2023-10-15T15:20:55.989 INFO:teuthology.orchestra.run.smithi019.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 192 kB/s | 122 kB 00:00 2023-10-15T15:21:01.181 INFO:teuthology.orchestra.run.smithi019.stdout:(10/10): ceph-test-18.0.0-6729.g984ba140.el9.x8 18 MB/s | 117 MB 00:06 2023-10-15T15:21:01.186 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:21:01.186 INFO:teuthology.orchestra.run.smithi019.stdout:Total 18 MB/s | 120 MB 00:06 2023-10-15T15:21:01.201 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:21:01.226 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:21:01.227 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:21:02.085 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:21:02.085 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:21:03.232 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:21:03.308 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-15T15:21:03.345 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-15T15:21:03.403 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-15T15:21:03.453 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-15T15:21:03.490 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-15T15:21:03.522 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-15T15:21:03.551 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-15T15:21:03.592 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-15T15:21:03.619 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 9/10 2023-10-15T15:21:20.804 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 9/10 2023-10-15T15:21:20.879 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 10/10 2023-10-15T15:21:28.985 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 10/10 2023-10-15T15:21:28.985 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 1/10 2023-10-15T15:21:28.986 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 2/10 2023-10-15T15:21:28.986 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-15T15:21:28.986 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-15T15:21:28.987 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-15T15:21:28.987 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-15T15:21:28.988 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-15T15:21:28.988 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-15T15:21:28.988 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-15T15:21:29.439 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-15T15:21:29.439 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:21:29.440 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:21:29.440 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:21:29.440 INFO:teuthology.orchestra.run.smithi019.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-15T15:21:29.440 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-15.el9.x86_64 2023-10-15T15:21:29.440 INFO:teuthology.orchestra.run.smithi019.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-15T15:21:29.441 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:21:29.441 INFO:teuthology.orchestra.run.smithi019.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-15T15:21:29.441 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-15T15:21:29.441 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-15T15:21:29.441 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-15T15:21:29.442 INFO:teuthology.orchestra.run.smithi019.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-15T15:21:29.442 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:21:29.442 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:21:29.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2023-10-15T15:21:30.353 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:05 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:21:30.512 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:21:30.515 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2023-10-15T15:21:30.515 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:21:30.515 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2023-10-15T15:21:30.515 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:21:30.515 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:18.0.0-6729.g984ba140.el9 ceph 6.5 k 2023-10-15T15:21:30.516 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:21:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:18.0.0-6729.g984ba140.el9 ceph 2.0 M 2023-10-15T15:21:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:18.0.0-6729.g984ba140.el9 ceph 1.5 M 2023-10-15T15:21:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 242 k 2023-10-15T15:21:30.516 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:18.0.0-6729.g984ba140.el9 ceph 8.8 M 2023-10-15T15:21:30.517 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:18.0.0-6729.g984ba140.el9 ceph 117 M 2023-10-15T15:21:30.517 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-15T15:21:30.517 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-15T15:21:30.517 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-15T15:21:30.517 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-15T15:21:30.518 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-15T15:21:30.518 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-15T15:21:30.518 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-15T15:21:30.518 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-15T15:21:30.518 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-15T15:21:30.519 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-15T15:21:30.519 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-15T15:21:30.519 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-15T15:21:30.519 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-15T15:21:30.519 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-15T15:21:30.520 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-15T15:21:30.520 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-15T15:21:30.520 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-15T15:21:30.520 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-15T15:21:30.520 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-15T15:21:30.521 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-15T15:21:30.521 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-15T15:21:30.521 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-15T15:21:30.521 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-15T15:21:30.521 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-15T15:21:30.522 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-15T15:21:30.522 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-15T15:21:30.522 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-15T15:21:30.522 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:21:30.522 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:21:30.523 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2023-10-15T15:21:30.523 INFO:teuthology.orchestra.run.smithi019.stdout:Install 33 Packages 2023-10-15T15:21:30.523 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:21:30.524 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 134 M 2023-10-15T15:21:30.524 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.4 G 2023-10-15T15:21:30.524 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:21:31.075 INFO:teuthology.orchestra.run.smithi019.stdout:(1/33): ceph-18.0.0-6729.g984ba140.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2023-10-15T15:21:31.800 INFO:teuthology.orchestra.run.smithi019.stdout:(2/33): ceph-mgr-18.0.0-6729.g984ba140.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-10-15T15:21:32.109 INFO:teuthology.orchestra.run.smithi019.stdout:(3/33): ceph-mds-18.0.0-6729.g984ba140.el9.x86_ 1.5 MB/s | 2.0 MB 00:01 2023-10-15T15:21:32.267 INFO:teuthology.orchestra.run.smithi019.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6729.g984b 1.5 MB/s | 242 kB 00:00 2023-10-15T15:21:32.376 INFO:teuthology.orchestra.run.smithi019.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2023-10-15T15:21:32.410 INFO:teuthology.orchestra.run.smithi019.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2023-10-15T15:21:32.443 INFO:teuthology.orchestra.run.smithi019.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-15T15:21:32.485 INFO:teuthology.orchestra.run.smithi019.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-10-15T15:21:32.527 INFO:teuthology.orchestra.run.smithi019.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-10-15T15:21:32.561 INFO:teuthology.orchestra.run.smithi019.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-10-15T15:21:32.594 INFO:teuthology.orchestra.run.smithi019.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-15T15:21:32.636 INFO:teuthology.orchestra.run.smithi019.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-10-15T15:21:32.828 INFO:teuthology.orchestra.run.smithi019.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 6.4 MB/s | 1.2 MB 00:00 2023-10-15T15:21:32.945 INFO:teuthology.orchestra.run.smithi019.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 437 kB/s | 51 kB 00:00 2023-10-15T15:21:33.187 INFO:teuthology.orchestra.run.smithi019.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 180 kB/s | 43 kB 00:00 2023-10-15T15:21:33.312 INFO:teuthology.orchestra.run.smithi019.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2023-10-15T15:21:33.430 INFO:teuthology.orchestra.run.smithi019.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2023-10-15T15:21:33.488 INFO:teuthology.orchestra.run.smithi019.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2023-10-15T15:21:33.550 INFO:teuthology.orchestra.run.smithi019.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 286 kB/s | 18 kB 00:00 2023-10-15T15:21:33.605 INFO:teuthology.orchestra.run.smithi019.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 423 kB/s | 23 kB 00:00 2023-10-15T15:21:33.664 INFO:teuthology.orchestra.run.smithi019.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 334 kB/s | 19 kB 00:00 2023-10-15T15:21:33.723 INFO:teuthology.orchestra.run.smithi019.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2023-10-15T15:21:33.840 INFO:teuthology.orchestra.run.smithi019.stdout:(23/33): ceph-mon-18.0.0-6729.g984ba140.el9.x86 3.2 MB/s | 8.8 MB 00:02 2023-10-15T15:21:33.865 INFO:teuthology.orchestra.run.smithi019.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 326 kB/s | 46 kB 00:00 2023-10-15T15:21:33.923 INFO:teuthology.orchestra.run.smithi019.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2023-10-15T15:21:33.982 INFO:teuthology.orchestra.run.smithi019.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2023-10-15T15:21:34.041 INFO:teuthology.orchestra.run.smithi019.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2023-10-15T15:21:34.091 INFO:teuthology.orchestra.run.smithi019.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2023-10-15T15:21:34.116 INFO:teuthology.orchestra.run.smithi019.stdout:(29/33): python3-more-itertools-8.12.0-2.el9.no 285 kB/s | 79 kB 00:00 2023-10-15T15:21:34.150 INFO:teuthology.orchestra.run.smithi019.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-10-15T15:21:34.200 INFO:teuthology.orchestra.run.smithi019.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2023-10-15T15:21:34.334 INFO:teuthology.orchestra.run.smithi019.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2023-10-15T15:21:48.733 INFO:teuthology.orchestra.run.smithi019.stdout:(33/33): ceph-osd-18.0.0-6729.g984ba140.el9.x86 6.9 MB/s | 117 MB 00:16 2023-10-15T15:21:48.738 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:21:48.739 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.3 MB/s | 134 MB 00:18 2023-10-15T15:21:48.771 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:21:48.814 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:21:48.815 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:21:49.840 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:21:49.841 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:21:51.718 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:21:51.785 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-15T15:21:51.818 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-15T15:21:51.851 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-15T15:21:51.947 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-15T15:21:52.032 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-15T15:21:52.087 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-15T15:21:52.125 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-15T15:21:52.205 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-15T15:21:52.255 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-10-15T15:21:52.293 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-15T15:21:52.352 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-15T15:21:52.434 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-15T15:21:52.648 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-15T15:21:52.697 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-15T15:21:52.736 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-15T15:21:52.771 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-15T15:21:52.809 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-15T15:21:52.888 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-15T15:21:52.919 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-15T15:21:52.952 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-15T15:21:52.994 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-15T15:21:53.028 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-15T15:21:53.123 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-15T15:21:53.159 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-15T15:21:53.236 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-15T15:21:53.386 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-15T15:21:53.517 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-15T15:21:53.698 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6729.g984ba140.el9. 28/33 2023-10-15T15:21:53.722 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 29/33 2023-10-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 29/33 2023-10-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:21:53.748 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-15T15:21:53.749 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-15T15:21:53.749 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-15T15:21:53.749 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:13.014 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 30/33 2023-10-15T15:22:13.042 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 30/33 2023-10-15T15:22:13.043 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:22:13.043 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-15T15:22:13.043 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-15T15:22:13.043 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-15T15:22:13.043 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:13.973 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 31/33 2023-10-15T15:22:13.997 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 31/33 2023-10-15T15:22:13.998 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:22:13.998 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-15T15:22:13.998 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-15T15:22:13.998 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-15T15:22:13.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:14.207 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 32/33 2023-10-15T15:22:14.231 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 32/33 2023-10-15T15:22:14.231 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:22:14.231 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-15T15:22:14.231 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-15T15:22:14.232 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-15T15:22:14.232 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:14.298 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:18.0.0-6729.g984ba140.el9.x86_64 33/33 2023-10-15T15:22:24.030 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:18.0.0-6729.g984ba140.el9.x86_64 33/33 2023-10-15T15:22:24.030 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:18.0.0-6729.g984ba140.el9.x86_64 1/33 2023-10-15T15:22:24.030 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 2/33 2023-10-15T15:22:24.031 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 3/33 2023-10-15T15:22:24.031 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 4/33 2023-10-15T15:22:24.031 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 5/33 2023-10-15T15:22:24.031 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6729.g984ba140.el9. 6/33 2023-10-15T15:22:24.031 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-15T15:22:24.032 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-15T15:22:24.032 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-15T15:22:24.032 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-15T15:22:24.032 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-15T15:22:24.032 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-15T15:22:24.032 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-15T15:22:24.033 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-10-15T15:22:24.033 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-15T15:22:24.033 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-15T15:22:24.033 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-15T15:22:24.033 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-15T15:22:24.034 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-15T15:22:24.034 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-15T15:22:24.034 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-15T15:22:24.034 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-15T15:22:24.034 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-15T15:22:24.035 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-15T15:22:24.035 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-15T15:22:24.035 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-15T15:22:24.035 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-15T15:22:24.036 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-15T15:22:24.036 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-15T15:22:24.037 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-15T15:22:24.037 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-15T15:22:24.037 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-15T15:22:24.934 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-15T15:22:24.934 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:24.935 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:24.935 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:22:24.935 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:22:24.935 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:22:24.935 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:24.936 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:22:24.936 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:22:24.936 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-15T15:22:24.936 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-15T15:22:24.936 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-15T15:22:24.936 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-15T15:22:24.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-15T15:22:24.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-15T15:22:24.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-15T15:22:24.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-15T15:22:24.937 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-15T15:22:24.938 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-15T15:22:24.938 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-15T15:22:24.938 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-15T15:22:24.938 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-15T15:22:24.938 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-15T15:22:24.939 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-15T15:22:24.939 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2023-10-15T15:22:24.939 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-15T15:22:24.939 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-15T15:22:24.939 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-15T15:22:24.939 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-15T15:22:24.940 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-15T15:22:24.940 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-15T15:22:24.940 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-15T15:22:24.940 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-15T15:22:24.941 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-15T15:22:24.941 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-15T15:22:24.941 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-15T15:22:24.941 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:24.941 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:25.220 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2023-10-15T15:22:25.834 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:00 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:25.927 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:22:25.992 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:25.992 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:22:25.993 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:26.061 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2023-10-15T15:22:26.667 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:01 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:26.811 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:26.812 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:26.813 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:22:26.813 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:26.813 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:26.813 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 269 k 2023-10-15T15:22:26.814 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:26.814 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:26.814 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:26.814 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:22:26.814 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:26.815 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 269 k 2023-10-15T15:22:26.815 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 264 k 2023-10-15T15:22:26.815 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:27.063 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-18.0.0-6729.g984ba140.el9.noarch.rpm 1.1 MB/s | 269 kB 00:00 2023-10-15T15:22:27.064 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:27.064 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.0 MB/s | 269 kB 00:00 2023-10-15T15:22:27.065 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:27.070 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:27.070 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:27.073 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:27.073 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:27.143 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:27.561 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:22:27.653 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:22:28.212 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:22:28.500 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:22:28.500 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:28.500 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:28.500 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:28.501 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:28.501 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:28.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2023-10-15T15:22:29.338 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:04 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:29.483 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:29.484 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:29.485 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-10-15T15:22:29.485 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:29.485 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:29.485 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6729.g984ba140.el9 ceph 144 k 2023-10-15T15:22:29.485 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:29.486 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:29.486 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:29.486 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:22:29.486 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:29.487 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 144 k 2023-10-15T15:22:29.487 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 437 k 2023-10-15T15:22:29.487 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:29.794 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-18.0.0-6729.g984ba1 467 kB/s | 144 kB 00:00 2023-10-15T15:22:29.795 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:29.795 INFO:teuthology.orchestra.run.smithi019.stdout:Total 464 kB/s | 144 kB 00:00 2023-10-15T15:22:29.796 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:29.803 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:29.803 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:29.858 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:29.858 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:30.112 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:30.178 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6729.g984ba140. 1/1 2023-10-15T15:22:30.203 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6729.g984ba140. 1/1 2023-10-15T15:22:30.204 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:22:30.204 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-15T15:22:30.205 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:31.223 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6729.g984ba140. 1/1 2023-10-15T15:22:31.224 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:31.224 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:31.224 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:22:31.224 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:31.225 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:31.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2023-10-15T15:22:32.103 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:07 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:32.197 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:22:32.248 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:32.249 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:22:32.249 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:32.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2023-10-15T15:22:32.922 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:07 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:33.066 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:33.067 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:33.067 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:22:33.067 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:33.067 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:33.068 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 1.8 M 2023-10-15T15:22:33.068 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:22:33.068 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 23 k 2023-10-15T15:22:33.068 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 15 k 2023-10-15T15:22:33.068 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-15T15:22:33.069 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-15T15:22:33.069 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-15T15:22:33.069 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2023-10-15T15:22:33.069 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-15T15:22:33.069 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:33.070 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:33.070 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:33.070 INFO:teuthology.orchestra.run.smithi019.stdout:Install 7 Packages 2023-10-15T15:22:33.070 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:33.071 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.1 M 2023-10-15T15:22:33.071 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 8.6 M 2023-10-15T15:22:33.071 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:33.382 INFO:teuthology.orchestra.run.smithi019.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6729.g984 110 kB/s | 23 kB 00:00 2023-10-15T15:22:33.407 INFO:teuthology.orchestra.run.smithi019.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6729.g984b 64 kB/s | 15 kB 00:00 2023-10-15T15:22:33.607 INFO:teuthology.orchestra.run.smithi019.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2023-10-15T15:22:33.666 INFO:teuthology.orchestra.run.smithi019.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 158 kB/s | 41 kB 00:00 2023-10-15T15:22:33.682 INFO:teuthology.orchestra.run.smithi019.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 410 kB/s | 31 kB 00:00 2023-10-15T15:22:33.825 INFO:teuthology.orchestra.run.smithi019.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2023-10-15T15:22:34.767 INFO:teuthology.orchestra.run.smithi019.stdout:(7/7): ceph-mgr-dashboard-18.0.0-6729.g984ba140 1.1 MB/s | 1.8 MB 00:01 2023-10-15T15:22:34.771 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:34.771 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.2 MB/s | 2.1 MB 00:01 2023-10-15T15:22:34.772 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:34.782 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:34.782 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:34.860 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:34.860 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:35.165 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:35.226 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-15T15:22:35.261 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-15T15:22:35.308 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-15T15:22:35.347 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-15T15:22:35.368 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6729.g984ba140.el9.n 5/7 2023-10-15T15:22:35.608 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6729.g984ba140.el9. 6/7 2023-10-15T15:22:35.637 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarc 7/7 2023-10-15T15:22:36.237 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarc 7/7 2023-10-15T15:22:36.237 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6729.g984ba140.el9. 1/7 2023-10-15T15:22:36.238 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarc 2/7 2023-10-15T15:22:36.238 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6729.g984ba140.el9.n 3/7 2023-10-15T15:22:36.238 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-15T15:22:36.238 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-15T15:22:36.239 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-15T15:22:36.564 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-15T15:22:36.565 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:36.565 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:36.565 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:36.565 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:36.566 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:36.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-15T15:22:36.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-15T15:22:36.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-15T15:22:36.566 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-15T15:22:36.566 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:36.567 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:36.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-15T15:22:37.391 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:12 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:37.538 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:37.539 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2023-10-15T15:22:37.539 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:22:37.539 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2023-10-15T15:22:37.539 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 7.4 M 2023-10-15T15:22:37.540 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:22:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-15T15:22:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-15T15:22:37.540 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-15T15:22:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-15T15:22:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-15T15:22:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-15T15:22:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-15T15:22:37.541 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-15T15:22:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-15T15:22:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-15T15:22:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-15T15:22:37.542 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:37.542 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:37.543 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2023-10-15T15:22:37.543 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2023-10-15T15:22:37.543 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:37.544 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 37 M 2023-10-15T15:22:37.544 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 209 M 2023-10-15T15:22:37.544 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:37.707 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 192 kB/s | 32 kB 00:00 2023-10-15T15:22:37.749 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2023-10-15T15:22:37.791 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 828 kB/s | 34 kB 00:00 2023-10-15T15:22:37.891 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2023-10-15T15:22:37.992 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.0 MB/s | 205 kB 00:00 2023-10-15T15:22:38.058 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-10-15T15:22:38.184 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): ceph-mgr-diskprediction-local-18.0.0-67 11 MB/s | 7.4 MB 00:00 2023-10-15T15:22:38.250 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2023-10-15T15:22:38.276 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.0 MB/s | 217 kB 00:00 2023-10-15T15:22:38.751 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2023-10-15T15:22:38.776 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 392 kB/s | 195 kB 00:00 2023-10-15T15:22:38.810 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2023-10-15T15:22:38.812 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:38.813 INFO:teuthology.orchestra.run.smithi019.stdout:Total 29 MB/s | 37 MB 00:01 2023-10-15T15:22:38.813 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:38.836 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:38.837 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:39.303 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:39.303 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:40.028 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:40.096 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-15T15:22:40.122 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-15T15:22:40.227 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-15T15:22:40.276 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-15T15:22:40.461 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-15T15:22:40.491 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-15T15:22:40.587 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-15T15:22:41.280 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-15T15:22:41.399 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-15T15:22:42.722 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-15T15:22:43.815 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-15T15:22:43.842 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba 12/12 2023-10-15T15:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba 12/12 2023-10-15T15:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba 1/12 2023-10-15T15:22:46.368 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-15T15:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-15T15:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-15T15:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-15T15:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-15T15:22:46.369 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-15T15:22:46.370 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-15T15:22:46.370 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-15T15:22:46.370 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-15T15:22:46.370 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-15T15:22:46.841 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-15T15:22:46.842 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:46.842 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:46.842 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:46.843 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-15T15:22:46.843 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-15T15:22:46.843 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-15T15:22:46.843 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-15T15:22:46.843 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-15T15:22:46.844 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-15T15:22:46.844 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-15T15:22:46.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-15T15:22:46.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-15T15:22:46.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-15T15:22:46.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-15T15:22:46.845 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:46.845 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:47.199 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2023-10-15T15:22:47.811 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:22 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:47.956 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:47.957 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-10-15T15:22:47.958 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:22:47.958 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-10-15T15:22:47.958 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:47.958 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 47 k 2023-10-15T15:22:47.959 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:22:47.959 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-15T15:22:47.959 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-15T15:22:47.959 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-15T15:22:47.960 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-15T15:22:47.960 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-15T15:22:47.960 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-15T15:22:47.960 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-15T15:22:47.960 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-15T15:22:47.960 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-15T15:22:47.961 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-15T15:22:47.961 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-15T15:22:47.961 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-15T15:22:47.961 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:47.961 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:47.962 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-10-15T15:22:47.962 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2023-10-15T15:22:47.962 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:47.963 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2023-10-15T15:22:47.963 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2023-10-15T15:22:47.963 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:48.243 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 159 kB/s | 18 kB 00:00 2023-10-15T15:22:48.268 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 24 kB 00:00 2023-10-15T15:22:48.318 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-10-15T15:22:48.343 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-15T15:22:48.369 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-15T15:22:48.394 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2023-10-15T15:22:48.486 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 164 kB/s | 15 kB 00:00 2023-10-15T15:22:48.503 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2023-10-15T15:22:48.520 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): ceph-mgr-rook-18.0.0-6729.g984ba140.el9 121 kB/s | 47 kB 00:00 2023-10-15T15:22:48.604 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-google-auth-2.23.2-1.el9.noarc 1.6 MB/s | 196 kB 00:00 2023-10-15T15:22:48.629 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2023-10-15T15:22:48.662 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2023-10-15T15:22:48.696 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2023-10-15T15:22:48.698 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:48.698 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2023-10-15T15:22:48.699 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:48.714 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:48.714 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:48.826 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:48.827 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:49.358 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:49.542 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-15T15:22:49.599 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-15T15:22:49.663 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-15T15:22:49.699 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-15T15:22:49.741 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-15T15:22:49.836 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-15T15:22:49.956 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 7/13 2023-10-15T15:22:50.018 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-15T15:22:50.777 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-15T15:22:50.830 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-15T15:22:50.864 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-15T15:22:50.910 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-15T15:22:50.933 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 13/13 2023-10-15T15:22:51.751 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 13/13 2023-10-15T15:22:51.751 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 1/13 2023-10-15T15:22:51.751 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-15T15:22:51.752 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-15T15:22:51.752 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-15T15:22:51.752 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-15T15:22:51.752 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-15T15:22:51.752 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-15T15:22:51.753 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-15T15:22:51.753 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-15T15:22:51.753 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 10/13 2023-10-15T15:22:51.753 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-15T15:22:51.753 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-15T15:22:52.131 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-15T15:22:52.131 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:52.131 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:52.131 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:52.132 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-15T15:22:52.132 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-15T15:22:52.132 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-15T15:22:52.132 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-15T15:22:52.132 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-15T15:22:52.133 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-15T15:22:52.133 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-15T15:22:52.133 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-15T15:22:52.133 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-15T15:22:52.133 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-15T15:22:52.134 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-15T15:22:52.134 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-15T15:22:52.134 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:52.134 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:52.403 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2023-10-15T15:22:53.013 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:28 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:53.159 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:53.160 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-10-15T15:22:53.160 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:22:53.161 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-10-15T15:22:53.161 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:53.161 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 132 k 2023-10-15T15:22:53.161 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:22:53.161 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-15T15:22:53.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-15T15:22:53.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-15T15:22:53.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-15T15:22:53.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-15T15:22:53.162 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:53.163 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:53.163 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2023-10-15T15:22:53.163 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2023-10-15T15:22:53.163 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:53.164 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.8 M 2023-10-15T15:22:53.164 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2023-10-15T15:22:53.164 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:53.690 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-10-15T15:22:53.865 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-10-15T15:22:53.923 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2023-10-15T15:22:53.974 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2023-10-15T15:22:53.999 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 431 kB/s | 58 kB 00:00 2023-10-15T15:22:54.024 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): ceph-mgr-cephadm-18.0.0-6729.g984ba140.e 267 kB/s | 132 kB 00:00 2023-10-15T15:22:54.026 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:54.027 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.8 MB/s | 6.8 MB 00:00 2023-10-15T15:22:54.027 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:54.037 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:54.037 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:54.132 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:54.133 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:54.440 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:54.616 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-15T15:22:54.660 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-15T15:22:55.222 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-15T15:22:55.315 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-15T15:22:55.382 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-15T15:22:55.413 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 6/6 2023-10-15T15:22:56.191 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 6/6 2023-10-15T15:22:56.191 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 1/6 2023-10-15T15:22:56.191 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-15T15:22:56.192 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-15T15:22:56.192 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-15T15:22:56.192 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-15T15:22:56.522 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-15T15:22:56.522 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:56.522 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:56.523 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:22:56.523 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-15T15:22:56.523 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-15T15:22:56.523 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-15T15:22:56.524 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-15T15:22:56.524 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-15T15:22:56.524 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:56.524 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:56.797 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2023-10-15T15:22:57.401 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:32 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:22:57.546 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:22:57.547 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:57.547 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:22:57.547 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:57.548 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:22:57.548 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:18.0.0-6729.g984ba140.el9 ceph 832 k 2023-10-15T15:22:57.548 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:22:57.548 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-15T15:22:57.548 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:57.549 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:22:57.549 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:22:57.549 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2023-10-15T15:22:57.549 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:57.549 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 910 k 2023-10-15T15:22:57.549 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 3.0 M 2023-10-15T15:22:57.550 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:22:57.685 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 575 kB/s | 79 kB 00:00 2023-10-15T15:22:58.118 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-18.0.0-6729.g984ba140.el9.x86_ 1.4 MB/s | 832 kB 00:00 2023-10-15T15:22:58.120 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:22:58.120 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.6 MB/s | 910 kB 00:00 2023-10-15T15:22:58.121 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:22:58.143 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:22:58.143 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:22:58.224 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:22:58.224 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:22:58.429 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:22:58.564 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-15T15:22:58.629 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 2/2 2023-10-15T15:22:59.304 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 2/2 2023-10-15T15:22:59.305 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/2 2023-10-15T15:22:59.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-15T15:22:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:59.622 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:22:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-15T15:22:59.622 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:22:59.623 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:22:59.931 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2023-10-15T15:23:00.539 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:35 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:00.685 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:00.686 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:23:00.687 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:00.687 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:23:00.687 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:18.0.0-6729.g984ba140.el9 ceph-noarch 262 k 2023-10-15T15:23:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:00.688 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:23:00.688 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:00.688 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:23:00.688 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:00.689 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 262 k 2023-10-15T15:23:00.689 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.2 M 2023-10-15T15:23:00.689 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:23:01.090 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-volume-18.0.0-6729.g984ba140.el9.noarch.rp 652 kB/s | 262 kB 00:00 2023-10-15T15:23:01.090 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:23:01.091 INFO:teuthology.orchestra.run.smithi019.stdout:Total 649 kB/s | 262 kB 00:00 2023-10-15T15:23:01.091 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:23:01.099 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:23:01.099 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:23:01.139 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:23:01.139 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:23:01.365 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:23:01.425 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:23:01.441 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:23:01.441 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:23:01.441 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-15T15:23:01.442 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:02.554 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 1/1 2023-10-15T15:23:02.555 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:02.555 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:23:02.555 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:18.0.0-6729.g984ba140.el9.noarch 2023-10-15T15:23:02.555 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:02.555 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:02.869 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2023-10-15T15:23:03.485 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:38 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:03.631 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:03.632 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:03.632 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-10-15T15:23:03.632 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:03.632 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:23:03.632 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 127 k 2023-10-15T15:23:03.633 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:03.633 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:23:03.633 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:03.633 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:23:03.634 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:03.634 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2023-10-15T15:23:03.634 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 456 k 2023-10-15T15:23:03.634 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:23:03.954 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-18.0.0-6729.g984ba140.el9.x86_64 395 kB/s | 127 kB 00:00 2023-10-15T15:23:03.954 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:23:03.955 INFO:teuthology.orchestra.run.smithi019.stdout:Total 393 kB/s | 127 kB 00:00 2023-10-15T15:23:03.955 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:23:03.960 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:23:03.960 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:23:04.018 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:23:04.019 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:23:04.176 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:23:04.271 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:04.725 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:05.059 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:05.060 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:23:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:23:05.060 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:05.060 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:05.308 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2023-10-15T15:23:05.914 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:40 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:06.008 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:06.060 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:06.061 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:06.061 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:06.139 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2023-10-15T15:23:06.749 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:41 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:06.894 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:06.895 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:06.895 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-10-15T15:23:06.895 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:06.895 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:23:06.896 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 31 k 2023-10-15T15:23:06.896 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:06.896 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:23:06.896 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:06.896 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:23:06.897 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:06.897 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 k 2023-10-15T15:23:06.897 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 139 k 2023-10-15T15:23:06.897 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:23:07.274 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-18.0.0-6729.g984ba140.el9.x86_6 82 kB/s | 31 kB 00:00 2023-10-15T15:23:07.275 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:23:07.275 INFO:teuthology.orchestra.run.smithi019.stdout:Total 82 kB/s | 31 kB 00:00 2023-10-15T15:23:07.275 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:23:07.279 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:23:07.316 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:23:07.317 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:23:07.406 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:23:07.526 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:07.932 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:08.227 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:08.227 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:08.227 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:23:08.227 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:23:08.228 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:08.228 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:08.397 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2023-10-15T15:23:09.006 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:44 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:09.098 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:09.151 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:09.152 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:09.152 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:09.235 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2023-10-15T15:23:09.841 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:44 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:09.933 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:09.985 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:09.986 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:09.986 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:10.076 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2023-10-15T15:23:10.678 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:45 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:10.771 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:10.823 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:10.824 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:10.824 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:10.905 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2023-10-15T15:23:11.511 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:46 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:11.605 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:11.660 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:11.661 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:11.661 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:11.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2023-10-15T15:23:12.345 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:47 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:12.440 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:12.492 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:12.493 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:12.493 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:12.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2023-10-15T15:23:13.186 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:48 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:13.279 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:18.0.0-6729.g984ba140.el9.x86_64 is already installed. 2023-10-15T15:23:13.332 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:13.333 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2023-10-15T15:23:13.334 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:13.422 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2023-10-15T15:23:14.028 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:49 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:14.174 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:14.175 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:14.175 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2023-10-15T15:23:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:23:14.176 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:18.0.0-6729.g984ba140.el9 ceph 87 k 2023-10-15T15:23:14.176 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:23:14.176 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:14.177 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:23:14.177 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:14.177 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 87 k 2023-10-15T15:23:14.177 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 252 k 2023-10-15T15:23:14.177 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:23:14.452 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-18.0.0-6729.g984ba140.el9.x86_64.rpm 314 kB/s | 87 kB 00:00 2023-10-15T15:23:14.453 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:23:14.453 INFO:teuthology.orchestra.run.smithi019.stdout:Total 313 kB/s | 87 kB 00:00 2023-10-15T15:23:14.453 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:23:14.460 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:23:14.460 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:23:14.519 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:23:14.519 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:23:14.658 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:23:14.744 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:15.190 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:15.483 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:15.483 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:15.483 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:23:15.484 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:23:15.484 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:15.484 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:15.754 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2023-10-15T15:23:16.361 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:51 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:16.512 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:16.513 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:16.513 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2023-10-15T15:23:16.514 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:16.514 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:23:16.514 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:18.0.0-6729.g984ba140.el9 ceph 3.0 M 2023-10-15T15:23:16.514 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:16.514 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:23:16.515 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:16.515 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:23:16.515 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:16.515 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.0 M 2023-10-15T15:23:16.515 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 13 M 2023-10-15T15:23:16.516 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:23:17.830 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-18.0.0-6729.g984ba140.el9.x86_64.rpm 2.3 MB/s | 3.0 MB 00:01 2023-10-15T15:23:17.831 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:23:17.831 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.3 MB/s | 3.0 MB 00:01 2023-10-15T15:23:17.832 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:23:17.843 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:23:17.843 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:23:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:23:17.923 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:23:18.374 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:23:18.431 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:18.456 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:18.456 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-15T15:23:18.457 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-15T15:23:18.457 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-15T15:23:18.457 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-15T15:23:18.457 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:19.683 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:19.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:19.684 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:23:19.684 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:23:19.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:19.684 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:19.971 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2023-10-15T15:23:20.579 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:55 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:23:20.726 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:23:20.727 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:20.727 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2023-10-15T15:23:20.727 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:20.727 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:23:20.727 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:18.0.0-6729.g984ba140.el9 ceph 168 k 2023-10-15T15:23:20.728 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:20.728 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:23:20.728 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:23:20.728 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2023-10-15T15:23:20.728 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:20.729 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 168 k 2023-10-15T15:23:20.729 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 532 k 2023-10-15T15:23:20.729 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:23:21.250 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-18.0.0-6729.g984ba140.el9.x86_64.rpm 322 kB/s | 168 kB 00:00 2023-10-15T15:23:21.252 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:23:21.252 INFO:teuthology.orchestra.run.smithi019.stdout:Total 321 kB/s | 168 kB 00:00 2023-10-15T15:23:21.252 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:23:21.260 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:23:21.260 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:23:21.320 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:23:21.320 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:23:21.468 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:23:21.568 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:22.111 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:22.442 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 1/1 2023-10-15T15:23:22.443 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:22.443 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:23:22.443 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:23:22.443 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:22.444 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:23:22.701 DEBUG:teuthology.parallel:result is None 2023-10-15T15:23:22.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=984ba140e1395377d07322ca248175a686ef49e1 2023-10-15T15:23:22.923 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-15T15:23:22.944 INFO:teuthology.orchestra.run.smithi019.stdout:18.0.0-6729.g984ba140.el9 2023-10-15T15:23:22.945 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6729.g984ba140.el9 2023-10-15T15:23:22.945 INFO:teuthology.task.install:The correct ceph version 18.0.0-6729.g984ba140 is installed. 2023-10-15T15:23:22.946 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-15T15:23:22.946 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:22.947 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-15T15:23:23.017 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-15T15:23:23.017 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:23.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2023-10-15T15:23:23.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-15T15:23:23.157 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-15T15:23:23.157 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:23.158 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-15T15:23:23.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-15T15:23:23.298 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-15T15:23:23.299 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:23.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2023-10-15T15:23:23.368 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-15T15:23:23.438 INFO:teuthology.run_tasks:Running task ceph... 2023-10-15T15:23:23.507 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-15T15:23:23.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2023-10-15T15:23:23.539 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-15T15:23:23.540 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-15T15:23:23.610 INFO:tasks.ceph:Creating extra log directories... 2023-10-15T15:23:23.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-15T15:23:23.688 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-15T15:23:23.689 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-15T15:23:23.689 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7428419', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '984ba140e1395377d07322ca248175a686ef49e1'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '9b943423e52ce7834e53a46d2315dd8f44efe018'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 4843, 'sha1': '984ba140e1395377d07322ca248175a686ef49e1', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_9b943423e52ce7834e53a46d2315dd8f44efe018/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '9b943423e52ce7834e53a46d2315dd8f44efe018', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKbU7DwDL1Koa9YrtiOTquFRHJu98yxKRBdS4efslIEShzuWkURIZGQP0dye2Ml8uIylp5u+U5BauRujswOGiI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '984ba140e1395377d07322ca248175a686ef49e1', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-15_14:39:28', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-15T15:23:23.690 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-15T15:23:23.736 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2023-10-15T15:23:23.807 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:23.808 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2023-10-15T15:23:23.863 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-15T15:23:23.863 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2023-10-15T15:23:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-15T15:23:23.918 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:23:23.919 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-15T15:23:23.919 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:23:23.919 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:23:23.919 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-10-15 15:23:18.707806319 +0000 2023-10-15T15:23:23.919 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-10-15 15:19:36.954634382 +0000 2023-10-15T15:23:23.920 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-10-15 15:19:36.954634382 +0000 2023-10-15T15:23:23.920 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-10-15 15:19:36.954634382 +0000 2023-10-15T15:23:23.920 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-15T15:23:23.987 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-10-15T15:23:23.987 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-10-15T15:23:23.987 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.00018133 s, 2.8 MB/s 2023-10-15T15:23:23.988 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-15T15:23:24.044 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2023-10-15T15:23:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-15T15:23:24.099 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:23:24.100 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-10-15T15:23:24.100 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:23:24.100 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:23:24.100 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-10-15 15:23:18.708806309 +0000 2023-10-15T15:23:24.100 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-10-15 15:19:37.445630021 +0000 2023-10-15T15:23:24.101 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-10-15 15:19:37.445630021 +0000 2023-10-15T15:23:24.101 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-10-15 15:19:37.445630021 +0000 2023-10-15T15:23:24.101 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-15T15:23:24.168 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-10-15T15:23:24.169 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-10-15T15:23:24.169 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000193187 s, 2.7 MB/s 2023-10-15T15:23:24.170 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-15T15:23:24.228 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2023-10-15T15:23:24.283 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-15T15:23:24.284 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:23:24.284 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-10-15T15:23:24.284 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:23:24.284 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:23:24.285 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-10-15 15:23:18.708806309 +0000 2023-10-15T15:23:24.285 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-10-15 15:19:37.945625580 +0000 2023-10-15T15:23:24.285 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-10-15 15:19:37.945625580 +0000 2023-10-15T15:23:24.285 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-10-15 15:19:37.945625580 +0000 2023-10-15T15:23:24.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-15T15:23:24.354 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-10-15T15:23:24.354 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-10-15T15:23:24.354 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000210732 s, 2.4 MB/s 2023-10-15T15:23:24.355 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-15T15:23:24.411 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2023-10-15T15:23:24.466 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-15T15:23:24.466 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-15T15:23:24.466 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-15T15:23:24.467 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-15T15:23:24.467 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2023-10-15T15:23:24.467 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2023-10-15 15:23:18.709806299 +0000 2023-10-15T15:23:24.467 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2023-10-15 15:19:38.448621112 +0000 2023-10-15T15:23:24.467 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2023-10-15 15:19:38.448621112 +0000 2023-10-15T15:23:24.468 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2023-10-15 15:19:38.448621112 +0000 2023-10-15T15:23:24.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-15T15:23:24.534 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2023-10-15T15:23:24.535 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2023-10-15T15:23:24.535 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000189205 s, 2.7 MB/s 2023-10-15T15:23:24.536 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-15T15:23:24.591 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:23:24.592 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-15T15:23:24.592 INFO:tasks.ceph:Generating config... 2023-10-15T15:23:24.593 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-15T15:23:24.593 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-15T15:23:24.594 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-10-15T15:23:24.594 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-15T15:23:24.594 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-15T15:23:24.595 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-10-15T15:23:24.595 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-15T15:23:24.595 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-15T15:23:24.595 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-15T15:23:24.596 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-10-15T15:23:24.596 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-10-15T15:23:24.596 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-10-15T15:23:24.596 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-10-15T15:23:24.597 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-15T15:23:24.597 INFO:tasks.ceph:[osd] err to stderr = True 2023-10-15T15:23:24.597 INFO:tasks.ceph:[osd] log flush on exit = True 2023-10-15T15:23:24.597 INFO:tasks.ceph:[osd] log to file = False 2023-10-15T15:23:24.598 INFO:tasks.ceph:[osd] log to stderr = True 2023-10-15T15:23:24.598 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-15T15:23:24.598 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-15T15:23:24.598 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-15T15:23:24.599 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-15T15:23:24.599 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-15T15:23:24.599 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-15T15:23:24.599 INFO:tasks.ceph:[global] auth client required = none 2023-10-15T15:23:24.600 INFO:tasks.ceph:[global] auth cluster required = none 2023-10-15T15:23:24.600 INFO:tasks.ceph:[global] auth service required = none 2023-10-15T15:23:24.600 INFO:tasks.ceph:[global] auth supported = none 2023-10-15T15:23:24.600 INFO:tasks.ceph:[global] debug asok = 0/0 2023-10-15T15:23:24.601 INFO:tasks.ceph:[global] debug auth = 0/0 2023-10-15T15:23:24.601 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-10-15T15:23:24.601 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-10-15T15:23:24.601 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-10-15T15:23:24.602 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-10-15T15:23:24.602 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-10-15T15:23:24.602 INFO:tasks.ceph:[global] debug client = 0/0 2023-10-15T15:23:24.603 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-10-15T15:23:24.603 INFO:tasks.ceph:[global] debug context = 0/0 2023-10-15T15:23:24.603 INFO:tasks.ceph:[global] debug crush = 0/0 2023-10-15T15:23:24.603 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-10-15T15:23:24.603 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-10-15T15:23:24.604 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-10-15T15:23:24.604 INFO:tasks.ceph:[global] debug filer = 0/0 2023-10-15T15:23:24.604 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-10-15T15:23:24.604 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-10-15T15:23:24.605 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-10-15T15:23:24.605 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-10-15T15:23:24.605 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-10-15T15:23:24.605 INFO:tasks.ceph:[global] debug journal = 0/0 2023-10-15T15:23:24.606 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-10-15T15:23:24.606 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-10-15T15:23:24.606 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-10-15T15:23:24.606 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-10-15T15:23:24.607 INFO:tasks.ceph:[global] debug mds = 0/0 2023-10-15T15:23:24.607 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-10-15T15:23:24.607 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-10-15T15:23:24.607 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-10-15T15:23:24.608 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-10-15T15:23:24.608 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-10-15T15:23:24.608 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-10-15T15:23:24.608 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-10-15T15:23:24.609 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-10-15T15:23:24.609 INFO:tasks.ceph:[global] debug mon = 0/0 2023-10-15T15:23:24.609 INFO:tasks.ceph:[global] debug monc = 0/0 2023-10-15T15:23:24.609 INFO:tasks.ceph:[global] debug ms = 0/0 2023-10-15T15:23:24.610 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-10-15T15:23:24.610 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-10-15T15:23:24.610 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-10-15T15:23:24.610 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-10-15T15:23:24.611 INFO:tasks.ceph:[global] debug osd = 0/0 2023-10-15T15:23:24.611 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-10-15T15:23:24.611 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-10-15T15:23:24.611 INFO:tasks.ceph:[global] debug rados = 0/0 2023-10-15T15:23:24.612 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-10-15T15:23:24.612 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-10-15T15:23:24.612 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-10-15T15:23:24.612 INFO:tasks.ceph:[global] debug refs = 0/0 2023-10-15T15:23:24.612 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-10-15T15:23:24.613 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-10-15T15:23:24.613 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-10-15T15:23:24.613 INFO:tasks.ceph:[global] debug striper = 0/0 2023-10-15T15:23:24.613 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-10-15T15:23:24.614 INFO:tasks.ceph:[global] debug timer = 0/0 2023-10-15T15:23:24.614 INFO:tasks.ceph:[global] debug tp = 0/0 2023-10-15T15:23:24.614 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-15T15:23:24.614 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-15T15:23:24.615 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-15T15:23:24.615 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-15T15:23:24.615 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-15T15:23:24.616 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-15T15:23:24.616 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-15T15:23:24.616 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-15T15:23:24.616 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-15T15:23:24.616 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-10-15T15:23:24.617 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-10-15T15:23:24.617 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-10-15T15:23:24.617 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-15T15:23:24.617 INFO:tasks.ceph:Setting up mon.a... 2023-10-15T15:23:24.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-15T15:23:24.697 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2023-10-15T15:23:24.700 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-15T15:23:24.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-15T15:23:24.783 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.19')] 2023-10-15T15:23:24.784 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-15T15:23:24.785 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:24.785 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-15T15:23:24.840 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:24.925 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:24.924+0000 7f76b3c38e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:24.926 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:24.924+0000 7f76b3c38e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:24.926 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:24.924+0000 7f76b3c38e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:24.927 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:24.927 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid c9e20d52-4b90-4a0c-80e4-c39b5a3b5114 2023-10-15T15:23:24.927 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = pacific 2023-10-15T15:23:24.928 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2023-10-15T15:23:24.928 INFO:teuthology.orchestra.run.smithi019.stdout:fsid c9e20d52-4b90-4a0c-80e4-c39b5a3b5114 2023-10-15T15:23:24.928 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2023-10-15T15:23:24.925338+0000 2023-10-15T15:23:24.928 INFO:teuthology.orchestra.run.smithi019.stdout:created 2023-10-15T15:23:24.925338+0000 2023-10-15T15:23:24.928 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 16 (pacific) 2023-10-15T15:23:24.929 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2023-10-15T15:23:24.929 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2023-10-15T15:23:24.929 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-15T15:23:24.929 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-15T15:23:24.944 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c9e20d52-4b90-4a0c-80e4-c39b5a3b5114... 2023-10-15T15:23:24.946 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-15T15:23:25.047 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2023-10-15T15:23:25.048 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2023-10-15T15:23:25.048 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-15T15:23:25.048 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = none 2023-10-15T15:23:25.048 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.049 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2023-10-15T15:23:25.049 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.049 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2023-10-15T15:23:25.049 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.050 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2023-10-15T15:23:25.050 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2023-10-15T15:23:25.050 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.050 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2023-10-15T15:23:25.050 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2023-10-15T15:23:25.051 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.051 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-15T15:23:25.051 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2023-10-15T15:23:25.052 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.052 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2023-10-15T15:23:25.052 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-15T15:23:25.052 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.053 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2023-10-15T15:23:25.053 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.053 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2023-10-15T15:23:25.053 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2023-10-15T15:23:25.054 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2023-10-15T15:23:25.054 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2023-10-15T15:23:25.054 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2023-10-15T15:23:25.054 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2023-10-15T15:23:25.055 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2023-10-15T15:23:25.055 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-15T15:23:25.055 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-15T15:23:25.055 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2023-10-15T15:23:25.055 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.056 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-15T15:23:25.056 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.056 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2023-10-15T15:23:25.056 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.057 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2023-10-15T15:23:25.057 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.057 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2023-10-15T15:23:25.057 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2023-10-15T15:23:25.058 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2023-10-15T15:23:25.058 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = 172.21.15.19 2023-10-15T15:23:25.058 INFO:teuthology.orchestra.run.smithi019.stdout: auth client required = none 2023-10-15T15:23:25.058 INFO:teuthology.orchestra.run.smithi019.stdout: auth cluster required = none 2023-10-15T15:23:25.059 INFO:teuthology.orchestra.run.smithi019.stdout: auth service required = none 2023-10-15T15:23:25.059 INFO:teuthology.orchestra.run.smithi019.stdout: debug asok = 0/0 2023-10-15T15:23:25.059 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 0/0 2023-10-15T15:23:25.059 INFO:teuthology.orchestra.run.smithi019.stdout: debug bdev = 0/0 2023-10-15T15:23:25.060 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 0/0 2023-10-15T15:23:25.060 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 0/0 2023-10-15T15:23:25.060 INFO:teuthology.orchestra.run.smithi019.stdout: debug buffer = 0/0 2023-10-15T15:23:25.060 INFO:teuthology.orchestra.run.smithi019.stdout: debug civetweb = 0/0 2023-10-15T15:23:25.060 INFO:teuthology.orchestra.run.smithi019.stdout: debug client = 0/0 2023-10-15T15:23:25.061 INFO:teuthology.orchestra.run.smithi019.stdout: debug compressor = 0/0 2023-10-15T15:23:25.061 INFO:teuthology.orchestra.run.smithi019.stdout: debug context = 0/0 2023-10-15T15:23:25.061 INFO:teuthology.orchestra.run.smithi019.stdout: debug crush = 0/0 2023-10-15T15:23:25.061 INFO:teuthology.orchestra.run.smithi019.stdout: debug crypto = 0/0 2023-10-15T15:23:25.061 INFO:teuthology.orchestra.run.smithi019.stdout: debug dpdk = 0/0 2023-10-15T15:23:25.062 INFO:teuthology.orchestra.run.smithi019.stdout: debug eventtrace = 0/0 2023-10-15T15:23:25.062 INFO:teuthology.orchestra.run.smithi019.stdout: debug filer = 0/0 2023-10-15T15:23:25.062 INFO:teuthology.orchestra.run.smithi019.stdout: debug filestore = 0/0 2023-10-15T15:23:25.062 INFO:teuthology.orchestra.run.smithi019.stdout: debug finisher = 0/0 2023-10-15T15:23:25.062 INFO:teuthology.orchestra.run.smithi019.stdout: debug fuse = 0/0 2023-10-15T15:23:25.063 INFO:teuthology.orchestra.run.smithi019.stdout: debug heartbeatmap = 0/0 2023-10-15T15:23:25.063 INFO:teuthology.orchestra.run.smithi019.stdout: debug javaclient = 0/0 2023-10-15T15:23:25.063 INFO:teuthology.orchestra.run.smithi019.stdout: debug journal = 0/0 2023-10-15T15:23:25.063 INFO:teuthology.orchestra.run.smithi019.stdout: debug journaler = 0/0 2023-10-15T15:23:25.064 INFO:teuthology.orchestra.run.smithi019.stdout: debug kstore = 0/0 2023-10-15T15:23:25.064 INFO:teuthology.orchestra.run.smithi019.stdout: debug leveldb = 0/0 2023-10-15T15:23:25.064 INFO:teuthology.orchestra.run.smithi019.stdout: debug lockdep = 0/0 2023-10-15T15:23:25.064 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds = 0/0 2023-10-15T15:23:25.065 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds balancer = 0/0 2023-10-15T15:23:25.065 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds locker = 0/0 2023-10-15T15:23:25.065 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds log = 0/0 2023-10-15T15:23:25.065 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds log expire = 0/0 2023-10-15T15:23:25.066 INFO:teuthology.orchestra.run.smithi019.stdout: debug mds migrator = 0/0 2023-10-15T15:23:25.066 INFO:teuthology.orchestra.run.smithi019.stdout: debug memdb = 0/0 2023-10-15T15:23:25.066 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 0/0 2023-10-15T15:23:25.066 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgrc = 0/0 2023-10-15T15:23:25.066 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 0/0 2023-10-15T15:23:25.067 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 0/0 2023-10-15T15:23:25.067 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2023-10-15T15:23:25.067 INFO:teuthology.orchestra.run.smithi019.stdout: debug objclass = 0/0 2023-10-15T15:23:25.067 INFO:teuthology.orchestra.run.smithi019.stdout: debug objectcacher = 0/0 2023-10-15T15:23:25.067 INFO:teuthology.orchestra.run.smithi019.stdout: debug objecter = 0/0 2023-10-15T15:23:25.068 INFO:teuthology.orchestra.run.smithi019.stdout: debug optracker = 0/0 2023-10-15T15:23:25.068 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 0/0 2023-10-15T15:23:25.068 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 0/0 2023-10-15T15:23:25.068 INFO:teuthology.orchestra.run.smithi019.stdout: debug perfcounter = 0/0 2023-10-15T15:23:25.069 INFO:teuthology.orchestra.run.smithi019.stdout: debug rados = 0/0 2023-10-15T15:23:25.069 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd = 0/0 2023-10-15T15:23:25.069 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd mirror = 0/0 2023-10-15T15:23:25.069 INFO:teuthology.orchestra.run.smithi019.stdout: debug rbd replay = 0/0 2023-10-15T15:23:25.069 INFO:teuthology.orchestra.run.smithi019.stdout: debug refs = 0/0 2023-10-15T15:23:25.070 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw = 0/0 2023-10-15T15:23:25.070 INFO:teuthology.orchestra.run.smithi019.stdout: debug rgw sync = 0/0 2023-10-15T15:23:25.070 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 0/0 2023-10-15T15:23:25.070 INFO:teuthology.orchestra.run.smithi019.stdout: debug striper = 0/0 2023-10-15T15:23:25.071 INFO:teuthology.orchestra.run.smithi019.stdout: debug throttle = 0/0 2023-10-15T15:23:25.071 INFO:teuthology.orchestra.run.smithi019.stdout: debug timer = 0/0 2023-10-15T15:23:25.071 INFO:teuthology.orchestra.run.smithi019.stdout: debug tp = 0/0 2023-10-15T15:23:25.071 INFO:teuthology.orchestra.run.smithi019.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-15T15:23:25.072 INFO:teuthology.orchestra.run.smithi019.stdout: ms client mode = crc 2023-10-15T15:23:25.072 INFO:teuthology.orchestra.run.smithi019.stdout: ms cluster mode = crc 2023-10-15T15:23:25.072 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon client mode = crc 2023-10-15T15:23:25.072 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon cluster mode = crc 2023-10-15T15:23:25.072 INFO:teuthology.orchestra.run.smithi019.stdout: ms mon service mode = crc 2023-10-15T15:23:25.073 INFO:teuthology.orchestra.run.smithi019.stdout: ms service mode = crc 2023-10-15T15:23:25.073 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = c9e20d52-4b90-4a0c-80e4-c39b5a3b5114 2023-10-15T15:23:25.073 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.073 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2023-10-15T15:23:25.073 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2023-10-15T15:23:25.074 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.074 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2023-10-15T15:23:25.074 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2023-10-15T15:23:25.074 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2023-10-15T15:23:25.074 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.075 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2023-10-15T15:23:25.075 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2023-10-15T15:23:25.075 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.076 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2023-10-15T15:23:25.076 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2023-10-15T15:23:25.076 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2023-10-15T15:23:25.076 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.076 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2023-10-15T15:23:25.077 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2023-10-15T15:23:25.077 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.077 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2023-10-15T15:23:25.077 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.077 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2023-10-15T15:23:25.078 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.078 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2023-10-15T15:23:25.078 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.078 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2023-10-15T15:23:25.079 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2023-10-15T15:23:25.079 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.079 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2023-10-15T15:23:25.079 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2023-10-15T15:23:25.079 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2023-10-15T15:23:25.080 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2023-10-15T15:23:25.080 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression algorithm = snappy 2023-10-15T15:23:25.080 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression mode = aggressive 2023-10-15T15:23:25.080 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2023-10-15T15:23:25.080 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore rocksdb cf = False 2023-10-15T15:23:25.081 INFO:teuthology.orchestra.run.smithi019.stdout: crimson seastar smp = 3 2023-10-15T15:23:25.081 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 20 2023-10-15T15:23:25.081 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 20 2023-10-15T15:23:25.081 INFO:teuthology.orchestra.run.smithi019.stdout: debug filestore = 0/0 2023-10-15T15:23:25.081 INFO:teuthology.orchestra.run.smithi019.stdout: debug journal = 0/0 2023-10-15T15:23:25.082 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2023-10-15T15:23:25.082 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 0/0 2023-10-15T15:23:25.082 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 10 2023-10-15T15:23:25.082 INFO:teuthology.orchestra.run.smithi019.stdout: err to stderr = True 2023-10-15T15:23:25.083 INFO:teuthology.orchestra.run.smithi019.stdout: log flush on exit = True 2023-10-15T15:23:25.083 INFO:teuthology.orchestra.run.smithi019.stdout: log to file = False 2023-10-15T15:23:25.083 INFO:teuthology.orchestra.run.smithi019.stdout: log to stderr = True 2023-10-15T15:23:25.083 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2023-10-15T15:23:25.084 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2023-10-15T15:23:25.084 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2023-10-15T15:23:25.084 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2023-10-15T15:23:25.084 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2023-10-15T15:23:25.084 INFO:teuthology.orchestra.run.smithi019.stdout: osd shutdown pgref assert = True 2023-10-15T15:23:25.085 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.085 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2023-10-15T15:23:25.085 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2023-10-15T15:23:25.085 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2023-10-15T15:23:25.086 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2023-10-15T15:23:25.086 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2023-10-15T15:23:25.086 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2023-10-15T15:23:25.086 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2023-10-15T15:23:25.087 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2023-10-15T15:23:25.087 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.087 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2023-10-15T15:23:25.087 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 0/0 2023-10-15T15:23:25.087 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 0/0 2023-10-15T15:23:25.088 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 0/0 2023-10-15T15:23:25.088 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2023-10-15T15:23:25.088 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2023-10-15T15:23:25.088 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2023-10-15T15:23:25.088 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2023-10-15T15:23:25.089 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2023-10-15T15:23:25.089 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2023-10-15T15:23:25.089 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2023-10-15T15:23:25.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.089 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-15T15:23:25.090 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2023-10-15T15:23:25.090 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2023-10-15T15:23:25.090 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.090 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2023-10-15T15:23:25.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-15T15:23:25.091 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-15T15:23:25.091 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.091 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2023-10-15T15:23:25.092 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2023-10-15T15:23:25.092 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.092 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2023-10-15T15:23:25.092 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default crimson = True 2023-10-15T15:23:25.093 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:25.093 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2023-10-15T15:23:25.093 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2023-10-15T15:23:25.093 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2023-10-15T15:23:25.093 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2023-10-15T15:23:25.094 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-15T15:23:25.094 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-15T15:23:25.094 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2023-10-15T15:23:25.095 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-15T15:23:25.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-15T15:23:25.161 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-15T15:23:25.162 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:25.162 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-15T15:23:25.177 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:25.177 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-15T15:23:25.233 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:23:25.234 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:25.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-15T15:23:25.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-15T15:23:25.320 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:25.321 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:25.376 INFO:tasks.ceph:Setting up mon nodes... 2023-10-15T15:23:25.376 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-15T15:23:25.377 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-15T15:23:25.486 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-15T15:23:25.489 INFO:tasks.ceph:Setting up mds nodes... 2023-10-15T15:23:25.489 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-15T15:23:25.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-15T15:23:25.542 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-15T15:23:25.561 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-15T15:23:25.561 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-15T15:23:25.561 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-15T15:23:25.594 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:23:25.595 INFO:tasks.ceph:role: osd.0 2023-10-15T15:23:25.595 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:23:25.595 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-15T15:23:25.674 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-15T15:23:25.674 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-15T15:23:25.675 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-15T15:23:25.675 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-15T15:23:25.675 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-15T15:23:25.675 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2023-10-15T15:23:25.676 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-15T15:23:25.676 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-15T15:23:25.676 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-15T15:23:25.676 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-15T15:23:26.183 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2023-10-15T15:23:26.185 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2023-10-15T15:23:26.186 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-15T15:23:26.340 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-15T15:23:26.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-15T15:23:26.451 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:23:26.452 INFO:tasks.ceph:role: osd.1 2023-10-15T15:23:26.452 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:23:26.452 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-15T15:23:26.529 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-15T15:23:26.530 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-15T15:23:26.530 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-15T15:23:26.530 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-15T15:23:26.530 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-15T15:23:26.531 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2023-10-15T15:23:26.531 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-15T15:23:26.531 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-15T15:23:26.531 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-15T15:23:26.531 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-15T15:23:26.987 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2023-10-15T15:23:26.990 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2023-10-15T15:23:26.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-15T15:23:27.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-15T15:23:27.125 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-15T15:23:27.196 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-15T15:23:27.196 INFO:tasks.ceph:role: osd.2 2023-10-15T15:23:27.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:23:27.197 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-15T15:23:27.274 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-15T15:23:27.275 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-15T15:23:27.275 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-15T15:23:27.275 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-15T15:23:27.275 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-15T15:23:27.276 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2023-10-15T15:23:27.276 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-15T15:23:27.276 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-15T15:23:27.276 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-15T15:23:27.276 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-15T15:23:27.771 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2023-10-15T15:23:27.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2023-10-15T15:23:27.774 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-15T15:23:27.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-15T15:23:27.969 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:28.528 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:23:28.529 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:28,528 seastar - Reactor backend: linux-aio 2023-10-15T15:23:28.549 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:28,548 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:23:28.549 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:28,548 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:23:28.549 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:28,548 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:23:28.550 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:28,548 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:23:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:==79814==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:23:28.557 INFO:teuthology.orchestra.run.smithi019.stderr:==79814==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd51322000; bottom 0x7f10c130d000; size: 0x00ec90015000 (1016028286976) 2023-10-15T15:23:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-10-15T15:23:28.558 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:23:28.583 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x22bc2dab 0x22c47ae6 0x22c48ba0 0x22c58d3b 0x22c5bd63 0x22caacb4 0x22c6f545 0x22c72b6e 0x22c8666f 0x22ca6311 0x22ca6b99 0x22c54c59 0x22c64279 0x22c65975 0x22c6634e 0x22c67439 0x22d39e28 0x22bb0cf9 0x22c782c4 0x22c79d0f 0x22c7aa78 0x22c428e4 0x22c41f3c 0x22973c88 0x249760b1 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ed6e1e8 0x1ed78355 0x1ebf5237 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:23:28.584 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-10-15T15:23:29.107 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:29.107 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-10-15T15:23:29.108 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-10-15T15:23:29.108 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-10-15T15:23:29.108 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:29.108 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:23:29.159 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:23:29.159 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:29,159 seastar - Reactor backend: linux-aio 2023-10-15T15:23:29.177 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:29,177 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:23:29.178 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:29,177 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:23:29.178 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:29,177 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:23:29.185 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:29,185 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:23:29.187 INFO:teuthology.orchestra.run.smithi019.stderr:==79812==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:23:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:==79812==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd51322000; bottom 0x7f10bf106000; size: 0x00ec9221c000 (1016063967232) 2023-10-15T15:23:29.194 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-10-15T15:23:29.194 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:23:29.220 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x228e6ed5 0x228f2da9 0x228f8df5 0x228fa7b4 0x24976b7e 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ebf6aed 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:23:29.220 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-10-15T15:23:29.280 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e6f12 0x261e955b 0x261eb36c 0x261edbd1 0x2601dc1f 0x1ecb029e 0x1ec25211 0x1ec2917d 0x2c9c2af5 2023-10-15T15:23:29.280 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb7c5d817 0xffffffffb7c59167 0xffffffffb7e000aa 2023-10-15T15:23:33.232 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:33.233 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-10-15T15:23:33.233 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-10-15T15:23:33.233 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-10-15T15:23:33.233 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:33.233 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:23:33.234 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid c9e20d52-4b90-4a0c-80e4-c39b5a3b5114 2023-10-15T15:23:33.269 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-15T15:23:33.300 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:33.843 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:23:33.844 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:33,843 seastar - Reactor backend: linux-aio 2023-10-15T15:23:33.864 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:33,863 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:23:33.864 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:33,863 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:23:33.864 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:33,863 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:23:33.865 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:33,863 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:23:33.866 INFO:teuthology.orchestra.run.smithi019.stderr:==79996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:23:33.872 INFO:teuthology.orchestra.run.smithi019.stderr:==79996==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffebf91f000; bottom 0x7fe231e63000; size: 0x001c8dabc000 (122635927552) 2023-10-15T15:23:33.872 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-10-15T15:23:33.872 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:23:33.898 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x22bc1023 0x22c4c6a3 0x22c4fae8 0x22c50036 0x22c44e42 0x22c46d0b 0x22c5f8b7 0x22c62848 0x22c8709c 0x22ca6311 0x22ca6b99 0x22c54c59 0x22c64279 0x22c65975 0x22c6634e 0x22c67439 0x22d39e28 0x22bb0cf9 0x22c782c4 0x22c79d0f 0x22c7aa78 0x22c428e4 0x22c41f3c 0x22973c88 0x249760b1 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ed6e1e8 0x1ed78355 0x1ebf5237 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:23:33.898 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-10-15T15:23:34.400 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:34.400 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-10-15T15:23:34.400 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-10-15T15:23:34.401 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-10-15T15:23:34.401 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:34.401 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:23:34.455 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:23:34.455 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:34,455 seastar - Reactor backend: linux-aio 2023-10-15T15:23:34.473 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:34,473 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:23:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:34,473 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:23:34.474 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:34,473 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:23:34.481 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:34,480 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:23:34.483 INFO:teuthology.orchestra.run.smithi019.stderr:==79994==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:23:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:==79994==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffebf91f000; bottom 0x7fe22fc5c000; size: 0x001c8fcc3000 (122671607808) 2023-10-15T15:23:34.489 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-10-15T15:23:34.490 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-10-15T15:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-10-15T15:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-10-15T15:23:38.525 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:38.526 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:23:38.526 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid c9e20d52-4b90-4a0c-80e4-c39b5a3b5114 2023-10-15T15:23:38.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-15T15:23:38.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:39.154 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:23:39.155 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,155 seastar - Reactor backend: linux-aio 2023-10-15T15:23:39.173 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,172 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:23:39.174 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,172 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:23:39.174 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,172 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:23:39.174 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,172 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:23:39.175 INFO:teuthology.orchestra.run.smithi019.stderr:==80178==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:23:39.181 INFO:teuthology.orchestra.run.smithi019.stderr:==80178==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcc863e000; bottom 0x7f51f13a7000; size: 0x00aad7297000 (733754257408) 2023-10-15T15:23:39.181 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-10-15T15:23:39.182 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:23:39.207 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0x22c9fe7d 0x22ca1e53 0x22ca2185 0x22ca383f 0x22ca66a6 0x22ca6b99 0x22c54c59 0x22c64279 0x22c65975 0x22c6634e 0x22c67439 0x22d39e28 0x22bb0cf9 0x22c782c4 0x22c79d0f 0x22c7aa78 0x22c428e4 0x22c41f3c 0x22973c88 0x249760b1 0x2497b7d9 0x2497dd74 0x2497ef4d 0x224ff745 0x2253afd2 0x22563349 0x2c5e11c5 0x2c67efe0 0x2c8a2dca 0x2c8a4c15 0x2c2fd244 0x2c2ff145 0x1ed6e1e8 0x1ed78355 0x1ebf5237 0x3feaf 0x3ff5f 0x1e6afa04 2023-10-15T15:23:39.208 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 2023-10-15T15:23:39.715 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:39.715 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-10-15T15:23:39.715 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-10-15T15:23:39.716 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-10-15T15:23:39.716 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:39.716 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:23:39.765 INFO:teuthology.orchestra.run.smithi019.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-15T15:23:39.765 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,765 seastar - Reactor backend: linux-aio 2023-10-15T15:23:39.783 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,783 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-15T15:23:39.783 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,783 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-15T15:23:39.784 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,783 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-15T15:23:39.791 INFO:teuthology.orchestra.run.smithi019.stderr:INFO 2023-10-15 15:23:39,790 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-15T15:23:39.793 INFO:teuthology.orchestra.run.smithi019.stderr:==80176==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-15T15:23:39.799 INFO:teuthology.orchestra.run.smithi019.stderr:==80176==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcc863e000; bottom 0x7f51ef1a0000; size: 0x00aad949e000 (733789937664) 2023-10-15T15:23:39.800 INFO:teuthology.orchestra.run.smithi019.stderr:False positive error reports may follow 2023-10-15T15:23:39.800 INFO:teuthology.orchestra.run.smithi019.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-15T15:23:39.884 INFO:teuthology.orchestra.run.smithi019.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63eeaa 0x2c640258 0x2c6417a6 0x2c641b2a 0x2c641c74 0x2c6420ca 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e6f12 0x261e955b 0x261eb36c 0x261edbd1 0x2601dc1f 0x1ecb029e 0x1ec25211 0x1ec2917d 0x2c9c2af5 2023-10-15T15:23:39.884 INFO:teuthology.orchestra.run.smithi019.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb7c5d817 0xffffffffb7c59167 0xffffffffb7e000aa 2023-10-15T15:23:43.870 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:43.870 INFO:teuthology.orchestra.run.smithi019.stderr:Suppressions used: 2023-10-15T15:23:43.870 INFO:teuthology.orchestra.run.smithi019.stderr: count bytes template 2023-10-15T15:23:43.870 INFO:teuthology.orchestra.run.smithi019.stderr: 1 8 MallocExtension::Register 2023-10-15T15:23:43.871 INFO:teuthology.orchestra.run.smithi019.stderr:----------------------------------------------------- 2023-10-15T15:23:43.871 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:23:43.871 INFO:teuthology.orchestra.run.smithi019.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid c9e20d52-4b90-4a0c-80e4-c39b5a3b5114 2023-10-15T15:23:43.906 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-15T15:23:43.936 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-15T15:23:43.937 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:43.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-15T15:23:44.008 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:44.008 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-15T15:23:44.078 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:44.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-15T15:23:44.148 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:44.148 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-15T15:23:44.218 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:44.219 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-15T15:23:44.273 INFO:tasks.ceph:Adding keys to all mons... 2023-10-15T15:23:44.273 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-15T15:23:44.342 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2023-10-15T15:23:44.342 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDtAyxl45f6HBAASWJfO3ewEpYmHUOD+eEHVg== 2023-10-15T15:23:44.342 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2023-10-15T15:23:44.343 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDxAyxlLjN5DhAAs6XymMZpD1cYIkUclhK5TQ== 2023-10-15T15:23:44.343 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2023-10-15T15:23:44.343 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQD2Ayxlm3wTIBAAhFOGDBymXv9ZAUdU34NI6w== 2023-10-15T15:23:44.343 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2023-10-15T15:23:44.343 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQD7Ayxlz6qcMhAAvK0wMk8fNcG0uj2kUVMQsw== 2023-10-15T15:23:44.344 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2023-10-15T15:23:44.344 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDtAyxlr9xIIBAAUmjfbdIyY5I6VvuPunjJfg== 2023-10-15T15:23:44.345 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-15T15:23:44.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-15T15:23:44.508 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-15T15:23:44.570 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-15T15:23:44.632 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-15T15:23:44.697 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-15T15:23:44.697 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-15T15:23:44.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-15T15:23:44.827 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:44.826+0000 7f9db1a35980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:44.828 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:44.826+0000 7f9db1a35980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:44.843 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-15T15:23:44.874 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-15T15:23:44.929 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-15T15:23:44.930 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-15T15:23:44.930 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-15T15:23:44.971 INFO:tasks.ceph.mon.a:Started 2023-10-15T15:23:44.972 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-15T15:23:44.972 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-15T15:23:44.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-15T15:23:44.974 INFO:tasks.ceph.mgr.x:Started 2023-10-15T15:23:44.975 INFO:tasks.ceph:Setting crush tunables to default 2023-10-15T15:23:44.975 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2023-10-15T15:23:45.060 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-10-15T15:23:45.058+0000 7f6f7f16c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.060 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-10-15T15:23:45.059+0000 7f6f7f16c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.061 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-10-15T15:23:45.059+0000 7f6f7f16c980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.061 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:45.060+0000 7fca14c14f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.094 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:45.085+0000 7f582402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.124 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:45.123+0000 7f582402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.194 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2023-10-15T15:23:45.205 INFO:tasks.ceph:check_enable_crimson: True 2023-10-15T15:23:45.205 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-15T15:23:45.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-15T15:23:45.265 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:45.263+0000 7fca14c14f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.265 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:45.263+0000 7fca14c14f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.316 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:45.311+0000 7f7e02b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.318 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:45.317+0000 7f7e02b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:45.399 INFO:tasks.ceph.mon.a.smithi019.stderr:2023-10-15T15:23:45.397+0000 7f6f773f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-15T15:23:45.399 INFO:tasks.ceph.mon.a.smithi019.stderr:Please be aware that this feature is experimental, untested, 2023-10-15T15:23:45.399 INFO:tasks.ceph.mon.a.smithi019.stderr:unsupported, and may result in data corruption, data loss, 2023-10-15T15:23:45.399 INFO:tasks.ceph.mon.a.smithi019.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-15T15:23:45.400 INFO:tasks.ceph.mon.a.smithi019.stderr:feature with important data. 2023-10-15T15:23:45.400 INFO:tasks.ceph.mon.a.smithi019.stderr: 2023-10-15T15:23:45.733 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:45.732+0000 7fca14c14f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-15T15:23:45.787 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:45.786+0000 7fca14c14f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-15T15:23:46.207 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-15T15:23:46.207 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:46.208 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-15T15:23:46.238 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:46.239 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-15T15:23:46.310 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:23:46.310 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-15T15:23:46.353 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.350+0000 7fca14c14f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-15T15:23:46.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new 55004121-1c2c-4257-a95b-c48663cc1805 0 2023-10-15T15:23:46.407 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.406+0000 7fca14c14f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-15T15:23:46.455 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.453+0000 7fca14c14f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-15T15:23:46.513 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.511+0000 7fca14c14f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-15T15:23:46.517 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:46.511+0000 7fa77a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:46.519 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:46.518+0000 7fa77a06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:46.558 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.556+0000 7fca14c14f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-15T15:23:46.597 INFO:teuthology.orchestra.run.smithi019.stdout:0 2023-10-15T15:23:46.608 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new 6678f46a-d69d-4957-b718-f9582614d333 1 2023-10-15T15:23:46.665 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.664+0000 7fca14c14f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-15T15:23:46.718 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:46.712+0000 7f1a4b95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:46.720 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:46.719+0000 7f1a4b95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:46.741 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.740+0000 7fca14c14f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-15T15:23:46.789 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.788+0000 7fca14c14f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-15T15:23:46.799 INFO:teuthology.orchestra.run.smithi019.stdout:1 2023-10-15T15:23:46.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd new 3382d65f-4f1f-416c-a3cf-fc02aecf98b6 2 2023-10-15T15:23:46.847 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.845+0000 7fca14c14f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-15T15:23:46.897 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:46.896+0000 7fca14c14f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-15T15:23:46.920 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:46.914+0000 7f421d66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:46.922 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:46.921+0000 7f421d66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:47.035 INFO:teuthology.orchestra.run.smithi019.stdout:2 2023-10-15T15:23:47.045 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-15T15:23:47.121 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-15T15:23:47.122 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-15T15:23:47.164 INFO:tasks.ceph.osd.0:Started 2023-10-15T15:23:47.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-15T15:23:47.201 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-15T15:23:47.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-15T15:23:47.245 INFO:tasks.ceph.osd.1:Started 2023-10-15T15:23:47.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-15T15:23:47.283 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-15T15:23:47.284 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-15T15:23:47.327 INFO:tasks.ceph.osd.2:Started 2023-10-15T15:23:47.328 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-15T15:23:47.359 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:47.358+0000 7fca14c14f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-15T15:23:47.527 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:47.525+0000 7fca14c14f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-15T15:23:47.543 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:47.535+0000 7f0f23d55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:47.547 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:47.546+0000 7f0f23d55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:47.679 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:47.678+0000 7fca14c14f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-15T15:23:47.699 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:47.700 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":6,"fsid":"c9e20d52-4b90-4a0c-80e4-c39b5a3b5114","created":"2023-10-15T15:23:45.115252+0000","modified":"2023-10-15T15:23:47.033961+0000","last_up_change":"0.000000","last_in_change":"2023-10-15T15:23:47.033961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"55004121-1c2c-4257-a95b-c48663cc1805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6678f46a-d69d-4957-b718-f9582614d333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3382d65f-4f1f-416c-a3cf-fc02aecf98b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:23:47.725 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-15T15:23:47.726 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-15T15:23:47.736 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:47.735+0000 7fca14c14f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-15T15:23:47.845 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:47.843+0000 7fca14c14f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-15T15:23:47.950 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:47.948+0000 7fca14c14f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-15T15:23:48.028 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-15T15:23:48.105 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:48.104+0000 7fca14c14f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-15T15:23:48.130 INFO:teuthology.misc.health.smithi019.stderr:2023-10-15T15:23:48.125+0000 7f1d8919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:48.131 INFO:teuthology.misc.health.smithi019.stderr:2023-10-15T15:23:48.130+0000 7f1d8919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:48.153 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:48.151+0000 7fca14c14f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-15T15:23:48.200 INFO:teuthology.misc.health.smithi019.stdout: 2023-10-15T15:23:48.201 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":6,"fsid":"c9e20d52-4b90-4a0c-80e4-c39b5a3b5114","created":"2023-10-15T15:23:45.115252+0000","modified":"2023-10-15T15:23:47.033961+0000","last_up_change":"0.000000","last_in_change":"2023-10-15T15:23:47.033961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"55004121-1c2c-4257-a95b-c48663cc1805","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6678f46a-d69d-4957-b718-f9582614d333","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3382d65f-4f1f-416c-a3cf-fc02aecf98b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:23:48.211 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-15T15:23:48.309 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:48.307+0000 7fca14c14f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-15T15:23:49.167 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:49.166+0000 7fca14c14f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-15T15:23:49.456 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:49.454+0000 7fca14c14f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-15T15:23:49.511 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:49.509+0000 7fca14c14f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-15T15:23:49.618 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:49.617+0000 7fca14c14f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-15T15:23:50.640 INFO:tasks.ceph.mgr.x.smithi019.stderr:2023-10-15T15:23:50.638+0000 7fc9b4d9c640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-15T15:23:54.514 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-15T15:23:54.625 INFO:teuthology.misc.health.smithi019.stderr:2023-10-15T15:23:54.621+0000 7fcd8aade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:54.627 INFO:teuthology.misc.health.smithi019.stderr:2023-10-15T15:23:54.626+0000 7fcd8aade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:54.820 INFO:teuthology.misc.health.smithi019.stdout: 2023-10-15T15:23:54.820 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":9,"fsid":"c9e20d52-4b90-4a0c-80e4-c39b5a3b5114","created":"2023-10-15T15:23:45.115252+0000","modified":"2023-10-15T15:23:54.646295+0000","last_up_change":"2023-10-15T15:23:52.640182+0000","last_in_change":"2023-10-15T15:23:47.033961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-15T15:23:53.693635+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55004121-1c2c-4257-a95b-c48663cc1805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4231732753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":4231732753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":4231732753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":4231732753}]},"public_addr":"172.21.15.19:6802/4231732753","cluster_addr":"172.21.15.19:6803/4231732753","heartbeat_back_addr":"172.21.15.19:6808/4231732753","heartbeat_front_addr":"172.21.15.19:6809/4231732753","state":["exists","up"]},{"osd":1,"uuid":"6678f46a-d69d-4957-b718-f9582614d333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":1752747341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":1752747341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":1752747341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":1752747341}]},"public_addr":"172.21.15.19:6804/1752747341","cluster_addr":"172.21.15.19:6805/1752747341","heartbeat_back_addr":"172.21.15.19:6810/1752747341","heartbeat_front_addr":"172.21.15.19:6811/1752747341","state":["exists","up"]},{"osd":2,"uuid":"3382d65f-4f1f-416c-a3cf-fc02aecf98b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3915628778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":3915628778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3915628778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":3915628778}]},"public_addr":"172.21.15.19:6806/3915628778","cluster_addr":"172.21.15.19:6807/3915628778","heartbeat_back_addr":"172.21.15.19:6812/3915628778","heartbeat_front_addr":"172.21.15.19:6813/3915628778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:23:54.831 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-15T15:23:54.831 INFO:tasks.ceph:Creating RBD pool 2023-10-15T15:23:54.831 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-15T15:23:54.942 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:54.935+0000 7f84dad04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:54.944 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:54.943+0000 7f84dad04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:55.654 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'rbd' created 2023-10-15T15:23:55.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-15T15:23:55.791 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:55.781+0000 7fbd4c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:55.793 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:55.792+0000 7fbd4c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:56.661 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-15T15:23:56.676 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-15T15:23:56.676 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-15T15:23:56.677 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-15T15:23:56.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-15T15:23:56.678 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-15T15:23:56.803 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:56.799+0000 7f289d206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:56.805 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:56.804+0000 7f289d206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.042 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:57.053 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":1369302547},{"type":"v1","addr":"172.21.15.19:6801","nonce":1369302547}]},"active_addr":"172.21.15.19:6801/1369302547","active_change":"2023-10-15T15:23:49.631715+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2670855265}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":1034736680}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.19:0","nonce":2316514726}]}]} 2023-10-15T15:23:57.056 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-15T15:23:57.056 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-15T15:23:57.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-15T15:23:57.167 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.163+0000 7efd0815e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.169 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.168+0000 7efd0815e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.353 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:57.354 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"fsid":"c9e20d52-4b90-4a0c-80e4-c39b5a3b5114","created":"2023-10-15T15:23:45.115252+0000","modified":"2023-10-15T15:23:56.657655+0000","last_up_change":"2023-10-15T15:23:52.640182+0000","last_in_change":"2023-10-15T15:23:47.033961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-15T15:23:53.693635+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-15T15:23:55.154723+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55004121-1c2c-4257-a95b-c48663cc1805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4231732753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":4231732753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":4231732753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":4231732753}]},"public_addr":"172.21.15.19:6802/4231732753","cluster_addr":"172.21.15.19:6803/4231732753","heartbeat_back_addr":"172.21.15.19:6808/4231732753","heartbeat_front_addr":"172.21.15.19:6809/4231732753","state":["exists","up"]},{"osd":1,"uuid":"6678f46a-d69d-4957-b718-f9582614d333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":1752747341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":1752747341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":1752747341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":1752747341}]},"public_addr":"172.21.15.19:6804/1752747341","cluster_addr":"172.21.15.19:6805/1752747341","heartbeat_back_addr":"172.21.15.19:6810/1752747341","heartbeat_front_addr":"172.21.15.19:6811/1752747341","state":["exists","up"]},{"osd":2,"uuid":"3382d65f-4f1f-416c-a3cf-fc02aecf98b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3915628778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":3915628778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3915628778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":3915628778}]},"public_addr":"172.21.15.19:6806/3915628778","cluster_addr":"172.21.15.19:6807/3915628778","heartbeat_back_addr":"172.21.15.19:6812/3915628778","heartbeat_front_addr":"172.21.15.19:6813/3915628778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:23:57.365 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-15T15:23:57.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-15T15:23:57.477 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.473+0000 7ff94702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.479 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.478+0000 7ff94702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:23:57.675 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"fsid":"c9e20d52-4b90-4a0c-80e4-c39b5a3b5114","created":"2023-10-15T15:23:45.115252+0000","modified":"2023-10-15T15:23:57.661254+0000","last_up_change":"2023-10-15T15:23:52.640182+0000","last_in_change":"2023-10-15T15:23:47.033961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-15T15:23:53.693635+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-15T15:23:55.154723+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55004121-1c2c-4257-a95b-c48663cc1805","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":4231732753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6803","nonce":4231732753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":4231732753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6809","nonce":4231732753}]},"public_addr":"172.21.15.19:6802/4231732753","cluster_addr":"172.21.15.19:6803/4231732753","heartbeat_back_addr":"172.21.15.19:6808/4231732753","heartbeat_front_addr":"172.21.15.19:6809/4231732753","state":["exists","up"]},{"osd":1,"uuid":"6678f46a-d69d-4957-b718-f9582614d333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":1752747341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6805","nonce":1752747341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":1752747341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6811","nonce":1752747341}]},"public_addr":"172.21.15.19:6804/1752747341","cluster_addr":"172.21.15.19:6805/1752747341","heartbeat_back_addr":"172.21.15.19:6810/1752747341","heartbeat_front_addr":"172.21.15.19:6811/1752747341","state":["exists","up"]},{"osd":2,"uuid":"3382d65f-4f1f-416c-a3cf-fc02aecf98b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3915628778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6807","nonce":3915628778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3915628778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6813","nonce":3915628778}]},"public_addr":"172.21.15.19:6806/3915628778","cluster_addr":"172.21.15.19:6807/3915628778","heartbeat_back_addr":"172.21.15.19:6812/3915628778","heartbeat_front_addr":"172.21.15.19:6813/3915628778","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-15T15:23:57.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-15T15:23:57.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-15T15:23:57.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-15T15:23:57.813 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.807+0000 7f0548d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.814 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.807+0000 7f0cbe19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.814 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.808+0000 7fe83379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.815 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.813+0000 7f0548d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.815 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.814+0000 7f0cbe19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.816 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.815+0000 7fe83379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.839 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-10-15T15:23:57.839 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-15T15:23:57.842 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-10-15T15:23:57.842 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-15T15:23:57.845 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-10-15T15:23:57.845 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-15T15:23:57.953 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.948+0000 7fdd4019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:57.955 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.954+0000 7fdd4019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:58.002 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:57.997+0000 7fa18d755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:58.004 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:58.003+0000 7fa18d755640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:58.021 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:58.016+0000 7f80d9e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:58.023 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:58.022+0000 7f80d9e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:58.168 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-10-15T15:23:58.182 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-15T15:23:58.193 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-10-15T15:23:58.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-15T15:23:58.213 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-10-15T15:23:58.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-15T15:23:59.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-15T15:23:59.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-15T15:23:59.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-15T15:23:59.312 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:59.307+0000 7f0166cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:59.315 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:59.314+0000 7f0166cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:59.357 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:59.352+0000 7f6ba126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:59.358 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:59.353+0000 7fdafa51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:59.359 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:59.358+0000 7f6ba126d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:59.360 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:23:59.359+0000 7fdafa51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:23:59.530 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-10-15T15:23:59.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-15T15:23:59.547 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-10-15T15:23:59.559 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-15T15:23:59.568 INFO:teuthology.orchestra.run.smithi019.stdout:34359738372 2023-10-15T15:23:59.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-15T15:24:00.544 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-15T15:24:00.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-15T15:24:00.582 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-15T15:24:00.677 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:00.669+0000 7f1d1d15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:00.679 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:00.678+0000 7f1d1d15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:00.715 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:00.710+0000 7fd2bcb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:00.717 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:00.715+0000 7fd2bcb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:00.725 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:00.720+0000 7f89270d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:00.729 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:00.728+0000 7f89270d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:00.890 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-10-15T15:24:00.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-15T15:24:00.902 DEBUG:teuthology.parallel:result is None 2023-10-15T15:24:00.908 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-10-15T15:24:00.920 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-15T15:24:00.921 DEBUG:teuthology.parallel:result is None 2023-10-15T15:24:00.924 INFO:teuthology.orchestra.run.smithi019.stdout:34359738373 2023-10-15T15:24:00.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-15T15:24:00.936 DEBUG:teuthology.parallel:result is None 2023-10-15T15:24:00.937 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-15T15:24:00.937 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-15T15:24:01.089 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:01.084+0000 7fc00d55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:01.090 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:01.089+0000 7fc00d55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:01.271 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:01.272 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2023-10-15T15:24:01.283 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-15T15:23:59.637528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59914,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.950095"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.713102+0000","last_change":"2023-10-15T15:23:56.743097+0000","last_active":"2023-10-15T15:23:57.713102+0000","last_peered":"2023-10-15T15:23:57.713102+0000","last_clean":"2023-10-15T15:23:57.713102+0000","last_became_active":"2023-10-15T15:23:56.739281+0000","last_became_peered":"2023-10-15T15:23:56.739281+0000","last_unstale":"2023-10-15T15:23:57.713102+0000","last_undegraded":"2023-10-15T15:23:57.713102+0000","last_fullsized":"2023-10-15T15:23:57.713102+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.700506+0000","last_change":"2023-10-15T15:23:56.782701+0000","last_active":"2023-10-15T15:23:57.700506+0000","last_peered":"2023-10-15T15:23:57.700506+0000","last_clean":"2023-10-15T15:23:57.700506+0000","last_became_active":"2023-10-15T15:23:56.770770+0000","last_became_peered":"2023-10-15T15:23:56.770770+0000","last_unstale":"2023-10-15T15:23:57.700506+0000","last_undegraded":"2023-10-15T15:23:57.700506+0000","last_fullsized":"2023-10-15T15:23:57.700506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.705588+0000","last_change":"2023-10-15T15:23:56.742797+0000","last_active":"2023-10-15T15:23:57.705588+0000","last_peered":"2023-10-15T15:23:57.705588+0000","last_clean":"2023-10-15T15:23:57.705588+0000","last_became_active":"2023-10-15T15:23:56.740936+0000","last_became_peered":"2023-10-15T15:23:56.740936+0000","last_unstale":"2023-10-15T15:23:57.705588+0000","last_undegraded":"2023-10-15T15:23:57.705588+0000","last_fullsized":"2023-10-15T15:23:57.705588+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.707810+0000","last_change":"2023-10-15T15:23:56.761579+0000","last_active":"2023-10-15T15:23:57.707810+0000","last_peered":"2023-10-15T15:23:57.707810+0000","last_clean":"2023-10-15T15:23:57.707810+0000","last_became_active":"2023-10-15T15:23:56.745432+0000","last_became_peered":"2023-10-15T15:23:56.745432+0000","last_unstale":"2023-10-15T15:23:57.707810+0000","last_undegraded":"2023-10-15T15:23:57.707810+0000","last_fullsized":"2023-10-15T15:23:57.707810+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.715272+0000","last_change":"2023-10-15T15:23:56.771348+0000","last_active":"2023-10-15T15:23:57.715272+0000","last_peered":"2023-10-15T15:23:57.715272+0000","last_clean":"2023-10-15T15:23:57.715272+0000","last_became_active":"2023-10-15T15:23:56.760672+0000","last_became_peered":"2023-10-15T15:23:56.760672+0000","last_unstale":"2023-10-15T15:23:57.715272+0000","last_undegraded":"2023-10-15T15:23:57.715272+0000","last_fullsized":"2023-10-15T15:23:57.715272+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.671145+0000","last_change":"2023-10-15T15:23:56.766290+0000","last_active":"2023-10-15T15:23:57.671145+0000","last_peered":"2023-10-15T15:23:57.671145+0000","last_clean":"2023-10-15T15:23:57.671145+0000","last_became_active":"2023-10-15T15:23:56.764265+0000","last_became_peered":"2023-10-15T15:23:56.764265+0000","last_unstale":"2023-10-15T15:23:57.671145+0000","last_undegraded":"2023-10-15T15:23:57.671145+0000","last_fullsized":"2023-10-15T15:23:57.671145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.671407+0000","last_change":"2023-10-15T15:23:56.750594+0000","last_active":"2023-10-15T15:23:57.671407+0000","last_peered":"2023-10-15T15:23:57.671407+0000","last_clean":"2023-10-15T15:23:57.671407+0000","last_became_active":"2023-10-15T15:23:56.748599+0000","last_became_peered":"2023-10-15T15:23:56.748599+0000","last_unstale":"2023-10-15T15:23:57.671407+0000","last_undegraded":"2023-10-15T15:23:57.671407+0000","last_fullsized":"2023-10-15T15:23:57.671407+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.699422+0000","last_change":"2023-10-15T15:23:56.749571+0000","last_active":"2023-10-15T15:23:57.699422+0000","last_peered":"2023-10-15T15:23:57.699422+0000","last_clean":"2023-10-15T15:23:57.699422+0000","last_became_active":"2023-10-15T15:23:56.747747+0000","last_became_peered":"2023-10-15T15:23:56.747747+0000","last_unstale":"2023-10-15T15:23:57.699422+0000","last_undegraded":"2023-10-15T15:23:57.699422+0000","last_fullsized":"2023-10-15T15:23:57.699422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.705953+0000","last_change":"2023-10-15T15:23:55.712177+0000","last_active":"2023-10-15T15:23:57.705953+0000","last_peered":"2023-10-15T15:23:57.705953+0000","last_clean":"2023-10-15T15:23:57.705953+0000","last_became_active":"2023-10-15T15:23:55.707175+0000","last_became_peered":"2023-10-15T15:23:55.707175+0000","last_unstale":"2023-10-15T15:23:57.705953+0000","last_undegraded":"2023-10-15T15:23:57.705953+0000","last_fullsized":"2023-10-15T15:23:57.705953+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:54.646295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:54.646295+0000","last_clean_scrub_stamp":"2023-10-15T15:23:54.646295+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-15T15:24:01.284 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-15T15:24:01.396 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:01.392+0000 7f1f6519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:01.398 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:01.396+0000 7f1f6519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:01.577 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:01.578 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2023-10-15T15:24:01.588 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-15T15:23:59.637528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59914,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.950095"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.713102+0000","last_change":"2023-10-15T15:23:56.743097+0000","last_active":"2023-10-15T15:23:57.713102+0000","last_peered":"2023-10-15T15:23:57.713102+0000","last_clean":"2023-10-15T15:23:57.713102+0000","last_became_active":"2023-10-15T15:23:56.739281+0000","last_became_peered":"2023-10-15T15:23:56.739281+0000","last_unstale":"2023-10-15T15:23:57.713102+0000","last_undegraded":"2023-10-15T15:23:57.713102+0000","last_fullsized":"2023-10-15T15:23:57.713102+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.700506+0000","last_change":"2023-10-15T15:23:56.782701+0000","last_active":"2023-10-15T15:23:57.700506+0000","last_peered":"2023-10-15T15:23:57.700506+0000","last_clean":"2023-10-15T15:23:57.700506+0000","last_became_active":"2023-10-15T15:23:56.770770+0000","last_became_peered":"2023-10-15T15:23:56.770770+0000","last_unstale":"2023-10-15T15:23:57.700506+0000","last_undegraded":"2023-10-15T15:23:57.700506+0000","last_fullsized":"2023-10-15T15:23:57.700506+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.705588+0000","last_change":"2023-10-15T15:23:56.742797+0000","last_active":"2023-10-15T15:23:57.705588+0000","last_peered":"2023-10-15T15:23:57.705588+0000","last_clean":"2023-10-15T15:23:57.705588+0000","last_became_active":"2023-10-15T15:23:56.740936+0000","last_became_peered":"2023-10-15T15:23:56.740936+0000","last_unstale":"2023-10-15T15:23:57.705588+0000","last_undegraded":"2023-10-15T15:23:57.705588+0000","last_fullsized":"2023-10-15T15:23:57.705588+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.707810+0000","last_change":"2023-10-15T15:23:56.761579+0000","last_active":"2023-10-15T15:23:57.707810+0000","last_peered":"2023-10-15T15:23:57.707810+0000","last_clean":"2023-10-15T15:23:57.707810+0000","last_became_active":"2023-10-15T15:23:56.745432+0000","last_became_peered":"2023-10-15T15:23:56.745432+0000","last_unstale":"2023-10-15T15:23:57.707810+0000","last_undegraded":"2023-10-15T15:23:57.707810+0000","last_fullsized":"2023-10-15T15:23:57.707810+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.715272+0000","last_change":"2023-10-15T15:23:56.771348+0000","last_active":"2023-10-15T15:23:57.715272+0000","last_peered":"2023-10-15T15:23:57.715272+0000","last_clean":"2023-10-15T15:23:57.715272+0000","last_became_active":"2023-10-15T15:23:56.760672+0000","last_became_peered":"2023-10-15T15:23:56.760672+0000","last_unstale":"2023-10-15T15:23:57.715272+0000","last_undegraded":"2023-10-15T15:23:57.715272+0000","last_fullsized":"2023-10-15T15:23:57.715272+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.671145+0000","last_change":"2023-10-15T15:23:56.766290+0000","last_active":"2023-10-15T15:23:57.671145+0000","last_peered":"2023-10-15T15:23:57.671145+0000","last_clean":"2023-10-15T15:23:57.671145+0000","last_became_active":"2023-10-15T15:23:56.764265+0000","last_became_peered":"2023-10-15T15:23:56.764265+0000","last_unstale":"2023-10-15T15:23:57.671145+0000","last_undegraded":"2023-10-15T15:23:57.671145+0000","last_fullsized":"2023-10-15T15:23:57.671145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.671407+0000","last_change":"2023-10-15T15:23:56.750594+0000","last_active":"2023-10-15T15:23:57.671407+0000","last_peered":"2023-10-15T15:23:57.671407+0000","last_clean":"2023-10-15T15:23:57.671407+0000","last_became_active":"2023-10-15T15:23:56.748599+0000","last_became_peered":"2023-10-15T15:23:56.748599+0000","last_unstale":"2023-10-15T15:23:57.671407+0000","last_undegraded":"2023-10-15T15:23:57.671407+0000","last_fullsized":"2023-10-15T15:23:57.671407+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.699422+0000","last_change":"2023-10-15T15:23:56.749571+0000","last_active":"2023-10-15T15:23:57.699422+0000","last_peered":"2023-10-15T15:23:57.699422+0000","last_clean":"2023-10-15T15:23:57.699422+0000","last_became_active":"2023-10-15T15:23:56.747747+0000","last_became_peered":"2023-10-15T15:23:56.747747+0000","last_unstale":"2023-10-15T15:23:57.699422+0000","last_undegraded":"2023-10-15T15:23:57.699422+0000","last_fullsized":"2023-10-15T15:23:57.699422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:55.652853+0000","last_clean_scrub_stamp":"2023-10-15T15:23:55.652853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-15T15:23:57.705953+0000","last_change":"2023-10-15T15:23:55.712177+0000","last_active":"2023-10-15T15:23:57.705953+0000","last_peered":"2023-10-15T15:23:57.705953+0000","last_clean":"2023-10-15T15:23:57.705953+0000","last_became_active":"2023-10-15T15:23:55.707175+0000","last_became_peered":"2023-10-15T15:23:55.707175+0000","last_unstale":"2023-10-15T15:23:57.705953+0000","last_undegraded":"2023-10-15T15:23:57.705953+0000","last_fullsized":"2023-10-15T15:23:57.705953+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-15T15:23:54.646295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-15T15:23:54.646295+0000","last_clean_scrub_stamp":"2023-10-15T15:23:54.646295+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-15T15:24:01.589 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-15T15:24:01.590 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-15T15:24:01.590 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-15T15:24:01.590 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-15T15:24:01.701 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:01.698+0000 7f8561664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:01.703 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:24:01.702+0000 7f8561664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:24:01.921 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:01.921 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-15T15:24:01.933 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-15T15:24:01.933 INFO:teuthology.run_tasks:Running task cbt... 2023-10-15T15:24:01.947 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi019.front.sepia.ceph.com', 'osds': ['ubuntu@smithi019.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi019.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi019.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi019.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-10-15T15:24:01.948 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-10-15T15:24:01.965 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:24:01.965 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-15T15:24:02.018 DEBUG:teuthology.orchestra.run.smithi019:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-10-15T15:24:02.073 INFO:teuthology.orchestra.run.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-10-15T15:24:02.482 INFO:tasks.cbt:Installing collectl 2023-10-15T15:24:02.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-10-15T15:24:02.594 INFO:teuthology.orchestra.run.smithi019.stderr:--2023-10-15 15:24:02-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-10-15T15:24:02.735 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2023-10-15T15:24:02.747 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-10-15T15:24:02.978 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-15T15:24:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlLAQSz62ARw7ydC39S-LX72hVy4hePyfeyg-tigzWtqzRJQJiI8lPvjLNILQzXC5BNar3eENhs4tuERNchiFQyUGIfw%3D%3D&use_mirror=cytranet&r= [following] 2023-10-15T15:24:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:--2023-10-15 15:24:02-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlLAQSz62ARw7ydC39S-LX72hVy4hePyfeyg-tigzWtqzRJQJiI8lPvjLNILQzXC5BNar3eENhs4tuERNchiFQyUGIfw%3D%3D&use_mirror=cytranet&r= 2023-10-15T15:24:02.979 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-10-15T15:24:03.050 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-10-15T15:24:03.289 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-15T15:24:03.289 INFO:teuthology.orchestra.run.smithi019.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-10-15T15:24:03.290 INFO:teuthology.orchestra.run.smithi019.stderr:--2023-10-15 15:24:03-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-10-15T15:24:03.301 INFO:teuthology.orchestra.run.smithi019.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2023-10-15T15:24:03.371 INFO:teuthology.orchestra.run.smithi019.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2023-10-15T15:24:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-15T15:24:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:Length: 620693 (606K) [application/x-gzip] 2023-10-15T15:24:03.702 INFO:teuthology.orchestra.run.smithi019.stderr:Saving to: ‘collectl.tar.gz’ 2023-10-15T15:24:03.703 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:24:03.772 INFO:teuthology.orchestra.run.smithi019.stderr: 0K .......... .......... .......... .......... .......... 8% 711K 1s 2023-10-15T15:24:03.841 INFO:teuthology.orchestra.run.smithi019.stderr: 50K .......... .......... .......... .......... .......... 16% 718K 1s 2023-10-15T15:24:03.842 INFO:teuthology.orchestra.run.smithi019.stderr: 100K .......... .......... .......... .......... .......... 24% 38.1M 0s 2023-10-15T15:24:03.911 INFO:teuthology.orchestra.run.smithi019.stderr: 150K .......... .......... .......... .......... .......... 32% 732K 0s 2023-10-15T15:24:03.912 INFO:teuthology.orchestra.run.smithi019.stderr: 200K .......... .......... .......... .......... .......... 41% 36.6M 0s 2023-10-15T15:24:03.913 INFO:teuthology.orchestra.run.smithi019.stderr: 250K .......... .......... .......... .......... .......... 49% 44.3M 0s 2023-10-15T15:24:03.948 INFO:teuthology.orchestra.run.smithi019.stderr: 300K .......... .......... .......... .......... .......... 57% 1.45M 0s 2023-10-15T15:24:03.981 INFO:teuthology.orchestra.run.smithi019.stderr: 350K .......... .......... .......... .......... .......... 65% 1.45M 0s 2023-10-15T15:24:03.982 INFO:teuthology.orchestra.run.smithi019.stderr: 400K .......... .......... .......... .......... .......... 74% 22.8M 0s 2023-10-15T15:24:03.984 INFO:teuthology.orchestra.run.smithi019.stderr: 450K .......... .......... .......... .......... .......... 82% 27.8M 0s 2023-10-15T15:24:03.988 INFO:teuthology.orchestra.run.smithi019.stderr: 500K .......... .......... .......... .......... .......... 90% 12.3M 0s 2023-10-15T15:24:03.990 INFO:teuthology.orchestra.run.smithi019.stderr: 550K .......... .......... .......... .......... .......... 98% 55.9M 0s 2023-10-15T15:24:03.990 INFO:teuthology.orchestra.run.smithi019.stderr: 600K ...... 100% 11.4T=0.3s 2023-10-15T15:24:03.990 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:24:03.991 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15 15:24:03 (2.05 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-10-15T15:24:03.991 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:24:04.035 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/ 2023-10-15T15:24:04.035 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/collectl.conf 2023-10-15T15:24:04.035 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ 2023-10-15T15:24:04.036 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/SlowProc.html 2023-10-15T15:24:04.036 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-10-15T15:24:04.036 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/WhySummary.html 2023-10-15T15:24:04.036 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Socket.html 2023-10-15T15:24:04.037 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/colmux.html 2023-10-15T15:24:04.037 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Tutorial.html 2023-10-15T15:24:04.037 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Gexpr.html 2023-10-15T15:24:04.037 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Import.html 2023-10-15T15:24:04.037 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/style.css 2023-10-15T15:24:04.037 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-10-15T15:24:04.038 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-10-15T15:24:04.038 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/FileNaming.html 2023-10-15T15:24:04.038 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/TheMath.html 2023-10-15T15:24:04.038 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Environmental.html 2023-10-15T15:24:04.038 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Process.html 2023-10-15T15:24:04.039 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-10-15T15:24:04.039 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/HiResTime.html 2023-10-15T15:24:04.039 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-10-15T15:24:04.039 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/slub.jpg 2023-10-15T15:24:04.039 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Lustre.html 2023-10-15T15:24:04.040 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Documentation.html 2023-10-15T15:24:04.040 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Inodes.html 2023-10-15T15:24:04.040 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Graphite.html 2023-10-15T15:24:04.040 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/InputFiles.html 2023-10-15T15:24:04.040 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Export.html 2023-10-15T15:24:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Interrupts.html 2023-10-15T15:24:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Startup.html 2023-10-15T15:24:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Performance.html 2023-10-15T15:24:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Messages.html 2023-10-15T15:24:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Logging.html 2023-10-15T15:24:04.041 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-detail.html 2023-10-15T15:24:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/CPUs.html 2023-10-15T15:24:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Matrix.html 2023-10-15T15:24:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-brief.html 2023-10-15T15:24:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-10-15T15:24:04.042 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Misc.html 2023-10-15T15:24:04.043 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Examples.html 2023-10-15T15:24:04.043 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OpenStack.html 2023-10-15T15:24:04.043 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-10-15T15:24:04.043 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-10-15T15:24:04.043 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-10-15T15:24:04.044 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-10-15T15:24:04.044 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Data.html 2023-10-15T15:24:04.044 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Exceptions.html 2023-10-15T15:24:04.044 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-10-15T15:24:04.044 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-10-15T15:24:04.045 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-10-15T15:24:04.045 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-10-15T15:24:04.045 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Playback.html 2023-10-15T15:24:04.045 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Hello.html 2023-10-15T15:24:04.045 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Sockets.html 2023-10-15T15:24:04.045 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Network.html 2023-10-15T15:24:04.046 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Infiniband.html 2023-10-15T15:24:04.046 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Memory.html 2023-10-15T15:24:04.046 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-10-15T15:24:04.046 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/Disks.html 2023-10-15T15:24:04.046 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-10-15T15:24:04.047 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/service/ 2023-10-15T15:24:04.047 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/service/collectl.service 2023-10-15T15:24:04.047 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/ 2023-10-15T15:24:04.047 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-suse 2023-10-15T15:24:04.047 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-debian 2023-10-15T15:24:04.048 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl-generic 2023-10-15T15:24:04.048 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/initd/collectl 2023-10-15T15:24:04.048 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/misc.ph 2023-10-15T15:24:04.048 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/gexpr.ph 2023-10-15T15:24:04.048 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/statsd.ph 2023-10-15T15:24:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/envrules.std 2023-10-15T15:24:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/hello.ph 2023-10-15T15:24:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vmsum.ph 2023-10-15T15:24:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/proctree.ph 2023-10-15T15:24:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/README-WINDOWS 2023-10-15T15:24:04.049 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/formatit.ph 2023-10-15T15:24:04.050 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/colmux 2023-10-15T15:24:04.050 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/INSTALL 2023-10-15T15:24:04.050 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/graphite.ph 2023-10-15T15:24:04.050 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/RELEASE-collectl 2023-10-15T15:24:04.052 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vmstat.ph 2023-10-15T15:24:04.052 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/README 2023-10-15T15:24:04.052 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/ 2023-10-15T15:24:04.052 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/collectl.1 2023-10-15T15:24:04.053 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/man1/colmux.1 2023-10-15T15:24:04.053 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/lexpr.ph 2023-10-15T15:24:04.053 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/client.pl 2023-10-15T15:24:04.053 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/collectl 2023-10-15T15:24:04.053 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/COPYING 2023-10-15T15:24:04.053 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/vnet.ph 2023-10-15T15:24:04.054 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/GPL 2023-10-15T15:24:04.054 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/UNINSTALL 2023-10-15T15:24:04.054 INFO:teuthology.orchestra.run.smithi019.stdout:collectl-4.3.1/ARTISTIC 2023-10-15T15:24:04.100 INFO:teuthology.orchestra.run.smithi019.stderr:./INSTALL: line 90: chkconfig: command not found 2023-10-15T15:24:04.102 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-10-15T15:24:04.709 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:03:39 ago on Sun 15 Oct 2023 03:20:25 PM UTC. 2023-10-15T15:24:04.803 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-10-15T15:24:04.806 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-10-15T15:24:04.863 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2023-10-15T15:24:04.864 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:24:04.864 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2023-10-15T15:24:04.864 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:24:04.864 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2023-10-15T15:24:04.865 INFO:teuthology.orchestra.run.smithi019.stdout: librbd-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 23 k 2023-10-15T15:24:04.865 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-10-15T15:24:04.865 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-10-15T15:24:04.865 INFO:teuthology.orchestra.run.smithi019.stdout: perf x86_64 5.14.0-375.el9 CentOS-AppStream 7.3 M 2023-10-15T15:24:04.865 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2023-10-15T15:24:04.866 INFO:teuthology.orchestra.run.smithi019.stdout: libradospp-devel x86_64 2:18.0.0-6729.g984ba140.el9 ceph 30 k 2023-10-15T15:24:04.866 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:04.866 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2023-10-15T15:24:04.866 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2023-10-15T15:24:04.866 INFO:teuthology.orchestra.run.smithi019.stdout:Install 5 Packages 2023-10-15T15:24:04.866 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:04.867 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 7.5 M 2023-10-15T15:24:04.867 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 11 M 2023-10-15T15:24:04.868 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2023-10-15T15:24:05.103 INFO:teuthology.orchestra.run.smithi019.stdout:(1/5): librbd-devel-18.0.0-6729.g984ba140.el9.x 126 kB/s | 23 kB 00:00 2023-10-15T15:24:05.253 INFO:teuthology.orchestra.run.smithi019.stdout:(2/5): perf-5.14.0-375.el9.x86_64.rpm 22 MB/s | 7.3 MB 00:00 2023-10-15T15:24:05.278 INFO:teuthology.orchestra.run.smithi019.stdout:(3/5): libradospp-devel-18.0.0-6729.g984ba140.e 84 kB/s | 30 kB 00:00 2023-10-15T15:24:05.412 INFO:teuthology.orchestra.run.smithi019.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 82 kB/s | 13 kB 00:00 2023-10-15T15:24:05.470 INFO:teuthology.orchestra.run.smithi019.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 324 kB/s | 118 kB 00:00 2023-10-15T15:24:05.472 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2023-10-15T15:24:05.473 INFO:teuthology.orchestra.run.smithi019.stdout:Total 12 MB/s | 7.5 MB 00:00 2023-10-15T15:24:05.473 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2023-10-15T15:24:05.496 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2023-10-15T15:24:05.496 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2023-10-15T15:24:05.660 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2023-10-15T15:24:05.661 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2023-10-15T15:24:05.966 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2023-10-15T15:24:06.032 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-10-15T15:24:06.062 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-10-15T15:24:06.093 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradospp-devel-2:18.0.0-6729.g984ba140.el9.x86_64 3/5 2023-10-15T15:24:06.441 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librbd-devel-2:18.0.0-6729.g984ba140.el9.x86_64 4/5 2023-10-15T15:24:06.568 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : perf-5.14.0-375.el9.x86_64 5/5 2023-10-15T15:24:07.611 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: perf-5.14.0-375.el9.x86_64 5/5 2023-10-15T15:24:07.611 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradospp-devel-2:18.0.0-6729.g984ba140.el9.x86_64 1/5 2023-10-15T15:24:07.611 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2/5 2023-10-15T15:24:07.611 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : perf-5.14.0-375.el9.x86_64 3/5 2023-10-15T15:24:07.612 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-10-15T15:24:07.936 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-10-15T15:24:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:07.937 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2023-10-15T15:24:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: libradospp-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:24:07.937 INFO:teuthology.orchestra.run.smithi019.stdout: librbd-devel-2:18.0.0-6729.g984ba140.el9.x86_64 2023-10-15T15:24:07.938 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-2.34-7.el9.x86_64 2023-10-15T15:24:07.938 INFO:teuthology.orchestra.run.smithi019.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-10-15T15:24:07.938 INFO:teuthology.orchestra.run.smithi019.stdout: perf-5.14.0-375.el9.x86_64 2023-10-15T15:24:07.938 INFO:teuthology.orchestra.run.smithi019.stdout: 2023-10-15T15:24:07.938 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2023-10-15T15:24:08.346 INFO:tasks.cbt:benchmark: radosbench 2023-10-15T15:24:08.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-10-15T15:24:08.380 DEBUG:teuthology.orchestra.run.smithi019:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-15T15:24:08.734 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - Settings.cluster: 2023-10-15T15:24:08.734 INFO:teuthology.orchestra.run.smithi019.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-10-15T15:24:08.734 INFO:teuthology.orchestra.run.smithi019.stderr: 'clients': ['ubuntu@smithi019.front.sepia.ceph.com'], 2023-10-15T15:24:08.735 INFO:teuthology.orchestra.run.smithi019.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-10-15T15:24:08.735 INFO:teuthology.orchestra.run.smithi019.stderr: 'head': 'ubuntu@smithi019.front.sepia.ceph.com', 2023-10-15T15:24:08.735 INFO:teuthology.orchestra.run.smithi019.stderr: 'is_teuthology': True, 2023-10-15T15:24:08.735 INFO:teuthology.orchestra.run.smithi019.stderr: 'iterations': 1, 2023-10-15T15:24:08.735 INFO:teuthology.orchestra.run.smithi019.stderr: 'mons': ['ubuntu@smithi019.front.sepia.ceph.com'], 2023-10-15T15:24:08.736 INFO:teuthology.orchestra.run.smithi019.stderr: 'osds': ['ubuntu@smithi019.front.sepia.ceph.com'], 2023-10-15T15:24:08.736 INFO:teuthology.orchestra.run.smithi019.stderr: 'osds_per_node': 3, 2023-10-15T15:24:08.736 INFO:teuthology.orchestra.run.smithi019.stderr: 'pid_dir': '/var/run/ceph', 2023-10-15T15:24:08.736 INFO:teuthology.orchestra.run.smithi019.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-10-15T15:24:08.737 INFO:teuthology.orchestra.run.smithi019.stderr: 'pgp_size': 256, 2023-10-15T15:24:08.737 INFO:teuthology.orchestra.run.smithi019.stderr: 'replication': 'replicated'}}, 2023-10-15T15:24:08.737 INFO:teuthology.orchestra.run.smithi019.stderr: 'rebuild_every_test': False, 2023-10-15T15:24:08.737 INFO:teuthology.orchestra.run.smithi019.stderr: 'rgws': {'ubuntu@smithi019.front.sepia.ceph.com': None}, 2023-10-15T15:24:08.737 INFO:teuthology.orchestra.run.smithi019.stderr: 'tmp_dir': '/tmp/cbt', 2023-10-15T15:24:08.738 INFO:teuthology.orchestra.run.smithi019.stderr: 'use_existing': True, 2023-10-15T15:24:08.738 INFO:teuthology.orchestra.run.smithi019.stderr: 'user': 'ubuntu'} 2023-10-15T15:24:08.755 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-10-15T15:24:08.756 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:24:08.757 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:24:08.758 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2023-10-15T15:24:08.760 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:24:08.761 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-15T15:24:08.762 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:24:08.762 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:24:08.762 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2023-10-15T15:24:08.765 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:24:08.766 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-15T15:24:08.766 INFO:teuthology.orchestra.run.smithi019.stderr:15:24:08 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-15T15:25:08.774 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.775 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-15T15:25:08.805 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.806 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-15T15:25:08.844 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.844 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-10-15T15:25:08.845 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.846 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-10-15T15:25:08.847 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.848 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.848 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2023-10-15T15:25:08.868 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.869 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/$(basename ${i}).smithi019.front.sepia.ceph.com ; done 2023-10-15T15:25:08.872 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.873 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-10-15T15:25:08.900 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - WARNING - cbt - sudo killall -9 rados 2023-10-15T15:25:08.900 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - WARNING - cbt - error 1 seen, continuing anyway... 2023-10-15T15:25:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2023-10-15T15:25:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-10-15T15:25:08.920 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.921 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-10-15T15:25:08.939 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.940 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.940 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2023-10-15T15:25:08.942 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.942 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-15T15:25:08.944 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.944 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.944 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2023-10-15T15:25:08.947 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.947 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-15T15:25:08.947 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-15T15:25:08.948 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.948 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:08.948 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-10-15T15:25:09.246 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:09 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:09.247 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:09 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:09.247 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:09 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-10-15T15:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:10 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:10 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-10-15T15:25:11.167 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:11 - INFO - cbt - Checking Health after pool creation. 2023-10-15T15:25:16.173 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:16 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:16.173 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:16 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:16.173 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-15T15:25:16.496 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:16 - INFO - cbt - Final Pool Health Check. 2023-10-15T15:25:21.502 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.502 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.503 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-15T15:25:21.822 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.822 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-15T15:25:21.839 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.839 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-15T15:25:21.872 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.872 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-10-15T15:25:21.873 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.873 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-10-15T15:25:21.875 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.875 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:21.875 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-15T15:25:22.213 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:22 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:22.213 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-15T15:25:22.821 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:22 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:22.822 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:22 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:22.822 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-10-15T15:25:23.124 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.124 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.124 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2023-10-15T15:25:23.178 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.178 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.178 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2023-10-15T15:25:23.947 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.948 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.948 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2023-10-15T15:25:23.950 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.951 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-15T15:25:23.953 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.953 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.953 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2023-10-15T15:25:23.957 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.958 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-15T15:25:23.958 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-15T15:25:23.958 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - INFO - cbt - Running radosbench write test. 2023-10-15T15:25:23.959 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.959 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.960 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2023-10-15T15:25:23.960 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.961 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:25:23.961 INFO:teuthology.orchestra.run.smithi019.stderr:15:25:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2023-10-15T15:26:25.062 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.063 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-15T15:26:25.089 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.089 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-15T15:26:25.122 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.123 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2023-10-15T15:26:25.124 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.124 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2023-10-15T15:26:25.125 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.126 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.126 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2023-10-15T15:26:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2023-10-15T15:26:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - get_fqdn_local()=smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/$(basename ${i}).smithi019.front.sepia.ceph.com ; done 2023-10-15T15:26:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - INFO - cbt - Convert results to json format. 2023-10-15T15:26:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:15:26:25 - DEBUG - cbt - Nodes : ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.562 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-10-15T15:26:25.574 DEBUG:teuthology.orchestra.run.smithi019:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-10-15T15:26:25.757 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-10-15T15:26:25.758 INFO:tasks.cbt_performance:reading cbt preferences 2023-10-15T15:26:25.759 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-10-15T15:26:25.817 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.817 INFO:teuthology.orchestra.run.smithi019.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi019.front.sepia.ceph.com 2023-10-15T15:26:25.818 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:26:25.818 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi019.front.sepia.ceph.com of=/dev/stdout 2023-10-15T15:26:25.837 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-15T15:26:25.838 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2023-10-15T15:26:25.892 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-15T15:26:25.893 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-15T15:26:25.893 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2023-10-15T15:26:25.894 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi019.front.sepia.ceph.com of=/dev/stdout 2023-10-15T15:26:25.947 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-15T15:26:25.948 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2023-10-15T15:26:26.002 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-15T15:26:26.003 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-15T15:26:26.011 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-15T15:26:26.018 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-15T15:26:26.019 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-15T15:26:26.029 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-15T15:26:26.159 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:26:26.151+0000 7f4ab4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:26:26.161 INFO:teuthology.orchestra.run.smithi019.stderr:2023-10-15T15:26:26.158+0000 7f4ab4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-15T15:26:26.362 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-15T15:26:26.363 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-15T15:26:26.363 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-15T15:26:26.363 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:26:31.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-15T15:26:32.465 INFO:tasks.ceph.osd.0:Stopped 2023-10-15T15:26:32.465 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-15T15:26:32.466 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:26:32.562 INFO:tasks.ceph.osd.1:Stopped 2023-10-15T15:26:32.563 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-15T15:26:32.563 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:26:36.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-15T15:26:36.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-15T15:26:38.665 INFO:tasks.ceph.osd.2:Stopped 2023-10-15T15:26:38.666 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-15T15:26:38.666 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-15T15:26:38.666 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:26:38.696 INFO:tasks.ceph.mgr.x:Stopped 2023-10-15T15:26:38.697 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-15T15:26:38.698 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-15T15:26:38.698 INFO:teuthology.orchestra.run:waiting for 300 2023-10-15T15:26:38.749 INFO:tasks.ceph.mon.a:Stopped 2023-10-15T15:26:38.750 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-15T15:26:38.750 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:26:38.859 INFO:teuthology.orchestra.run.smithi019.stdout:2023-10-15T15:25:54.232022+0000 mon.a (mon.0) 156 : cluster [WRN] Health check failed: Degraded data redundancy: 174/9500 objects degraded (1.832%), 10 pgs degraded (PG_DEGRADED) 2023-10-15T15:26:38.859 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-15T15:26:38.860 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:26:38.896 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:26:38.972 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-15T15:26:39.047 INFO:teuthology.orchestra.run.smithi019.stdout:2023-10-15T15:25:54.232022+0000 mon.a (mon.0) 156 : cluster [WRN] Health check failed: Degraded data redundancy: 174/9500 objects degraded (1.832%), 10 pgs degraded (PG_DEGRADED) 2023-10-15T15:26:39.047 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:39.047 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-15T15:26:39.447 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:39.447 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-15T15:26:39.554 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:39.554 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-15T15:26:39.646 INFO:tasks.ceph:Archiving mon data... 2023-10-15T15:26:39.658 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419/data/mon.a.tgz 2023-10-15T15:26:39.658 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2023-10-15T15:26:39.754 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.LPItZ3dpyH 2023-10-15T15:26:39.755 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LPItZ3dpyH 2023-10-15T15:26:39.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.LPItZ3dpyH 2023-10-15T15:26:39.963 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.LPItZ3dpyH is 179KB 2023-10-15T15:26:39.998 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.LPItZ3dpyH 2023-10-15T15:26:40.015 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-15T15:26:40.015 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-15T15:26:40.087 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-15T15:26:40.087 DEBUG:teuthology.orchestra.run.smithi019:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-15T15:26:40.170 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-15T15:26:40.175 INFO:tasks.ceph:Archiving crash dumps... 2023-10-15T15:26:40.177 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419/remote/smithi019/crash 2023-10-15T15:26:40.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-15T15:26:40.246 INFO:tasks.ceph:Compressing logs... 2023-10-15T15:26:40.247 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-15T15:26:40.325 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-15T15:26:40.326 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83283.log 2023-10-15T15:26:40.326 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79328.log 2023-10-15T15:26:40.326 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.83283.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82109.log 2023-10-15T15:26:40.331 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81572.log 2023-10-15T15:26:40.331 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.79328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81983.log 2023-10-15T15:26:40.332 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83485.log 2023-10-15T15:26:40.332 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82403.log 2023-10-15T15:26:40.332 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82275.log 2023-10-15T15:26:40.332 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82022.log 2023-10-15T15:26:40.332 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83345.log 2023-10-15T15:26:40.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82235.log 2023-10-15T15:26:40.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81897.log 2023-10-15T15:26:40.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-15T15:26:40.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81670.log 2023-10-15T15:26:40.333 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80937.log 2023-10-15T15:26:40.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80897.log 2023-10-15T15:26:40.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82149.log 2023-10-15T15:26:40.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82445.log 2023-10-15T15:26:40.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81896.log 2023-10-15T15:26:40.334 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83397.log 2023-10-15T15:26:40.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-15T15:26:40.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83484.log 2023-10-15T15:26:40.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83721.log 2023-10-15T15:26:40.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81731.log 2023-10-15T15:26:40.335 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82274.log 2023-10-15T15:26:40.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81628.log 2023-10-15T15:26:40.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82361.log 2023-10-15T15:26:40.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82023.log 2023-10-15T15:26:40.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80787.log 2023-10-15T15:26:40.336 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82148.log 2023-10-15T15:26:40.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81773.log 2023-10-15T15:26:40.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80699.log 2023-10-15T15:26:40.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81135.log 2023-10-15T15:26:40.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83420.log 2023-10-15T15:26:40.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-15T15:26:40.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81815.log 2023-10-15T15:26:40.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83374.log 2023-10-15T15:26:40.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81165.log 2023-10-15T15:26:40.340 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-15T15:26:40.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81895.log 2023-10-15T15:26:40.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-15T15:26:40.341 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-15T15:26:40.342 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80977.log 2023-10-15T15:26:40.346 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.81572.log: /var/log/ceph/ceph-client.admin.81628.log: /var/log/ceph/ceph-client.admin.83485.log: /var/log/ceph/ceph-client.admin.82361.log: /var/log/ceph/ceph-client.admin.82023.log: /var/log/ceph/ceph-client.admin.80787.log: /var/log/ceph/ceph-client.admin.81773.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.81135.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.83484.log: /var/log/ceph/ceph-client.admin.83721.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.0.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.83485.log.gz 2023-10-15T15:26:40.346 INFO:teuthology.orchestra.run.smithi019.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-15T15:26:40.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: 66.9% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83721.log.gz 2023-10-15T15:26:40.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.82109.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83283.log.gz 2023-10-15T15:26:40.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.82403.log: /var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph-client.admin.83484.log.gz 2023-10-15T15:26:40.347 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.83345.log: /var/log/ceph/ceph-client.admin.81983.log: /var/log/ceph/ceph-client.admin.82275.log: /var/log/ceph/ceph-client.admin.82022.log: /var/log/ceph/ceph-client.admin.80699.log: /var/log/ceph/ceph-client.admin.83374.log: /var/log/ceph/ceph-client.admin.83420.log: /var/log/ceph/ceph-client.admin.82148.log: /var/log/ceph/ceph-client.admin.80977.log: /var/log/ceph/ceph-client.admin.82235.log: 85.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-15T15:26:40.348 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.81897.log: /var/log/ceph/ceph-client.admin.81670.log: /var/log/ceph/ceph-client.admin.81815.log: /var/log/ceph/ceph-client.admin.81165.log: /var/log/ceph/ceph-client.admin.80937.log: /var/log/ceph/ceph-client.admin.80897.log: /var/log/ceph/ceph-client.admin.82149.log: /var/log/ceph/ceph-client.admin.82445.log: /var/log/ceph/ceph-client.admin.81896.log: /var/log/ceph/ceph-client.admin.81895.log: /var/log/ceph/ceph-client.admin.83397.log: /var/log/ceph/ceph-client.admin.82274.log: /var/log/ceph/ceph-client.admin.81731.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80787.log.gz 2023-10-15T15:26:40.348 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80699.log.gz 2023-10-15T15:26:40.348 INFO:teuthology.orchestra.run.smithi019.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79328.log.gz 2023-10-15T15:26:40.356 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83345.log.gz 2023-10-15T15:26:40.357 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83374.log.gz 2023-10-15T15:26:40.357 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83397.log.gz 2023-10-15T15:26:40.357 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83420.log.gz 2023-10-15T15:26:40.403 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81135.log.gz 2023-10-15T15:26:40.404 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81165.log.gz 2023-10-15T15:26:40.408 INFO:teuthology.orchestra.run.smithi019.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-15T15:26:40.409 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80977.log.gz 2023-10-15T15:26:40.425 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80937.log.gz 2023-10-15T15:26:40.426 INFO:teuthology.orchestra.run.smithi019.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-15T15:26:40.450 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80897.log.gz 2023-10-15T15:26:40.503 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81572.log.gz 2023-10-15T15:26:40.503 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81628.log.gz 2023-10-15T15:26:40.503 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81670.log.gz 2023-10-15T15:26:40.504 INFO:teuthology.orchestra.run.smithi019.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.81731.log.gz 2023-10-15T15:26:40.504 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81773.log.gz 2023-10-15T15:26:40.504 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81815.log.gz 2023-10-15T15:26:40.504 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81897.log.gz 2023-10-15T15:26:40.504 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81895.log.gz -- replaced with /var/log/ceph/ceph-client.admin.81896.log.gz 2023-10-15T15:26:40.505 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:26:40.505 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81983.log.gz 2023-10-15T15:26:40.505 INFO:teuthology.orchestra.run.smithi019.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.82022.log.gz 2023-10-15T15:26:40.505 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82023.log.gz 2023-10-15T15:26:40.506 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82109.log.gz 2023-10-15T15:26:40.506 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82148.log.gz 2023-10-15T15:26:40.506 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82149.log.gz 2023-10-15T15:26:40.506 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82235.log.gz 2023-10-15T15:26:40.506 INFO:teuthology.orchestra.run.smithi019.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82274.log.gz 2023-10-15T15:26:40.507 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82275.log.gz 2023-10-15T15:26:40.507 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82361.log.gz 2023-10-15T15:26:40.507 INFO:teuthology.orchestra.run.smithi019.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82403.log.gz 2023-10-15T15:26:40.507 INFO:teuthology.orchestra.run.smithi019.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82445.log.gz 2023-10-15T15:26:41.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-15T15:26:41.648 INFO:teuthology.orchestra.run.smithi019.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-15T15:26:41.676 INFO:teuthology.orchestra.run.smithi019.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-15T15:26:41.764 INFO:teuthology.orchestra.run.smithi019.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-15T15:26:41.766 INFO:teuthology.orchestra.run.smithi019.stderr: 2023-10-15T15:26:41.767 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m1.464s 2023-10-15T15:26:41.767 INFO:teuthology.orchestra.run.smithi019.stderr:user 0m2.778s 2023-10-15T15:26:41.767 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m0.139s 2023-10-15T15:26:41.767 INFO:tasks.ceph:Archiving logs... 2023-10-15T15:26:41.768 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419/remote/smithi019/log 2023-10-15T15:26:41.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-15T15:26:42.307 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-15T15:26:42.318 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-15T15:26:42.318 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-15T15:26:42.376 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-15T15:26:42.386 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-15T15:26:42.386 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-15T15:26:42.404 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2023-10-15T15:26:42.475 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-15T15:26:42.476 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2023-10-15T15:26:42.476 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 +1342us[+1344us] +/- 85ms 2023-10-15T15:26:42.476 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 -952us[ -950us] +/- 76ms 2023-10-15T15:26:42.477 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv03.front.sepia.ceph.com 3 6 377 5 -210us[ -208us] +/- 69ms 2023-10-15T15:26:42.477 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-15T15:26:42.492 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-15T15:26:42.501 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-15T15:26:42.501 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-15T15:26:42.510 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-15T15:26:42.714 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-15T15:26:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:26:42.751 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-15T15:26:42.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-15T15:26:42.770 INFO:teuthology.task.internal:Duration was 731.882257 seconds 2023-10-15T15:26:42.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-15T15:26:42.780 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-15T15:26:42.780 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-15T15:26:42.928 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-15T15:26:43.212 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-15T15:26:43.212 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2023-10-15T15:26:43.213 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-15T15:26:43.233 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-15T15:26:43.234 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-15T15:26:43.306 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-15T15:26:43.307 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-15T15:26:43.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-15T15:26:43.461 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-15T15:26:43.461 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-15T15:26:43.510 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-15T15:26:43.521 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-15T15:26:43.583 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2023-10-15T15:26:43.617 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-15T15:26:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-15T15:26:43.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-15T15:26:43.642 INFO:teuthology.task.internal:Transferring archived files... 2023-10-15T15:26:43.643 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419/remote/smithi019 2023-10-15T15:26:43.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-15T15:26:43.752 INFO:teuthology.task.internal:Removing archive directory... 2023-10-15T15:26:43.753 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-15T15:26:43.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-15T15:26:43.823 INFO:teuthology.task.internal:Not uploading archives. 2023-10-15T15:26:43.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-15T15:26:43.833 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-15T15:26:43.833 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-15T15:26:43.864 INFO:teuthology.orchestra.run.smithi019.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 15 15:26 /home/ubuntu/cephtest 2023-10-15T15:26:43.865 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-15T15:26:43.876 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-15T15:26:43.897 INFO:teuthology.nuke:Checking targets against current locks 2023-10-15T15:26:43.923 DEBUG:teuthology.nuke:shortname: smithi019 2023-10-15T15:26:43.924 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-15T15:26:43.948 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-15_14:39:28-crimson-rados-main-distro-crimson-smithi/7428419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-15 15:06:19.569325', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGKbU7DwDL1Koa9YrtiOTquFRHJu98yxKRBdS4efslIEShzuWkURIZGQP0dye2Ml8uIylp5u+U5BauRujswOGiI='} 2023-10-15T15:26:44.000 INFO:teuthology.orchestra.console.smithi019:Power off 2023-10-15T15:26:44.001 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-15T15:26:44.023 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2023-10-15T15:26:44.024 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:26:44.047 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2023-10-15T15:26:48.049 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:26:48.159 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2023-10-15T15:26:52.161 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-15T15:26:52.271 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2023-10-15T15:26:52.372 INFO:teuthology.orchestra.console.smithi019:Power off completed 2023-10-15T15:26:52.477 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=main&var-job_id_selected=7428419 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} duration: 731.8822572231293 failure_reason: '"2023-10-15T15:25:54.232022+0000 mon.a (mon.0) 156 : cluster [WRN] Health check failed: Degraded data redundancy: 174/9500 objects degraded (1.832%), 10 pgs degraded (PG_DEGRADED)" in cluster log' flavor: crimson owner: scheduled_matan@teuthology success: false 2023-10-15T15:26:52.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-15T15:26:52.569 INFO:teuthology.run:FAIL