2023-12-04T08:46:11.571 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T08:46:11.571 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} --archive /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785 --verbose --name matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi --owner scheduled_matan@teuthology -- /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785/orig.config.yaml 2023-12-04T08:46:11.613 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T08:46:11.693 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785 branch: wip-matanb-crimson-submit-error-cleanup-testing description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} email: null first_in_suite: false job_id: '7476785' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-submit-error-cleanup-testing ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: ebb1a6e3649b44fd77bc9418b26b48a91c567607 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: ebb1a6e3649b44fd77bc9418b26b48a91c567607 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: e89bb53732569ae4a8d78945935b6e5cb35576be owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9400 sha1: ebb1a6e3649b44fd77bc9418b26b48a91c567607 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_e89bb53732569ae4a8d78945935b6e5cb35576be/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: e89bb53732569ae4a8d78945935b6e5cb35576be targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkY88tiDoPV86zWCdQX5Liqvop9xun1VX6pXHPUmWPRNToKKXsI8rrVjM1HldcpZVjNXGnqlF6ItHv0oahRovM= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_08:39:13 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T08:46:11.693 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_e89bb53732569ae4a8d78945935b6e5cb35576be/qa; will attempt to use it 2023-12-04T08:46:11.694 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_e89bb53732569ae4a8d78945935b6e5cb35576be/qa/tasks 2023-12-04T08:46:11.694 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T08:46:11.696 INFO:teuthology.task.internal:Checking packages... 2023-12-04T08:46:11.718 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'ebb1a6e3649b44fd77bc9418b26b48a91c567607' 2023-12-04T08:46:11.718 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T08:46:11.718 INFO:teuthology.packaging:ref: None 2023-12-04T08:46:11.719 INFO:teuthology.packaging:tag: None 2023-12-04T08:46:11.719 INFO:teuthology.packaging:branch: wip-matanb-crimson-submit-error-cleanup-testing 2023-12-04T08:46:11.719 INFO:teuthology.packaging:sha1: ebb1a6e3649b44fd77bc9418b26b48a91c567607 2023-12-04T08:46:11.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-submit-error-cleanup-testing 2023-12-04T08:46:11.861 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7693.gebb1a6e3 2023-12-04T08:46:11.861 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T08:46:11.868 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T08:46:11.868 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T08:46:11.875 INFO:teuthology.task.internal:Saving configuration 2023-12-04T08:46:11.891 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T08:46:11.897 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T08:46:11.920 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 08:39:36.848332', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkY88tiDoPV86zWCdQX5Liqvop9xun1VX6pXHPUmWPRNToKKXsI8rrVjM1HldcpZVjNXGnqlF6ItHv0oahRovM='} 2023-12-04T08:46:11.920 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T08:46:11.927 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-04T08:46:11.927 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T08:46:11.967 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd0bf1ac9d0>, signals=[15]) 2023-12-04T08:46:11.967 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T08:46:11.973 INFO:teuthology.task.internal:Opening connections... 2023-12-04T08:46:11.973 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T08:46:11.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T08:46:12.073 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T08:46:12.080 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2023-12-04T08:46:12.143 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2023-12-04T08:46:12.143 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2023-12-04T08:46:12.200 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-04T08:46:12.201 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2023-12-04T08:46:12.201 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-04T08:46:12.201 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-04T08:46:12.201 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-04T08:46:12.201 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-12-04T08:46:12.227 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T08:46:14.133 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T08:46:14.142 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T08:46:14.143 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2023-12-04T08:46:14.161 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T08:46:14.167 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T08:46:14.167 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2023-12-04T08:46:14.224 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T08:46:14.256 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T08:46:14.270 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-04T08:46:14.270 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-04T08:46:14.271 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T08:46:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T08:46:14.288 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-12-04T08:46:14.345 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2023-12-04T08:46:14.346 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2023-12-04T08:46:14.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2023-12-04T08:46:16.572 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2023-12-04T08:46:18.047 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2023-12-04T08:46:19.705 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 7.9 MB 00:01 2023-12-04T08:46:31.414 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 1.7 MB/s | 18 MB 00:10 2023-12-04T08:46:35.731 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 5.8 MB 00:00 2023-12-04T08:46:37.148 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2023-12-04T08:46:42.291 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 20 MB 00:04 2023-12-04T08:46:47.199 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2023-12-04T08:46:49.623 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-04T08:46:49.623 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-04T08:46:49.706 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T08:46:49.712 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T08:46:49.712 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-12-04T08:46:49.712 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2023-12-04T08:46:49.713 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T08:46:49.714 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 96 M 2023-12-04T08:46:49.714 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 121 M 2023-12-04T08:46:49.714 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T08:46:51.567 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 3.5 MB/s | 5.6 MB 00:01 2023-12-04T08:46:52.667 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 7.6 MB/s | 20 MB 00:02 2023-12-04T08:46:54.276 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 8.8 MB/s | 38 MB 00:04 2023-12-04T08:46:58.101 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 5.0 MB/s | 32 MB 00:06 2023-12-04T08:46:58.101 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T08:46:58.101 INFO:teuthology.orchestra.run.smithi006.stdout:Total 11 MB/s | 96 MB 00:08 2023-12-04T08:46:58.993 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T08:46:59.453 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T08:46:59.453 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T08:47:00.824 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T08:47:00.824 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T08:47:03.627 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T08:47:04.292 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-04T08:47:04.426 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T08:47:07.310 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T08:47:07.496 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T08:47:12.451 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T08:47:12.571 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-04T08:47:17.552 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T08:47:48.426 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T08:47:48.470 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-04T08:47:48.874 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-04T08:47:48.874 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-04T08:47:48.874 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-04T08:47:48.874 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-04T08:47:49.308 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T08:47:49.309 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T08:47:49.797 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T08:47:49.986 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-04T08:47:49.987 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-04T08:47:49.987 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-04T08:47:49.987 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-04T08:47:49.987 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-04T08:47:49.987 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-04T08:47:49.987 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2023-12-04T08:47:49.987 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2023-12-04T08:47:50.111 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-04T08:47:50.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-04T08:47:50.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2023-12-04T08:47:50.154 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2023-12-04T08:47:50.154 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2023-12-04T08:47:50.154 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2023-12-04T08:47:50.155 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T08:47:50.156 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2023-12-04T08:47:50.157 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-04T08:47:50.157 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-388.el9.x86_64 2023-12-04T08:47:50.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-04T08:47:50.807 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2023-12-04T08:47:52.373 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-04T08:47:52.426 INFO:teuthology.orchestra.run.smithi006.stderr:done 2023-12-04T08:47:52.429 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-12-04T08:47:52.444 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.jfpZtoWgFp 2023-12-04T08:47:52.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jfpZtoWgFp 2023-12-04T08:47:52.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.jfpZtoWgFp 2023-12-04T08:47:52.666 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.jfpZtoWgFp is 6KB 2023-12-04T08:47:52.715 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.jfpZtoWgFp 2023-12-04T08:47:52.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2023-12-04T08:47:52.799 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-04T08:47:52.799 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-04T08:47:52.799 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-04T08:47:52.799 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-04T08:47:52.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-04T08:47:52.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2023-12-04T08:48:22.986 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-04T08:48:22.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-04T08:48:22.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T08:49:23.047 DEBUG:teuthology.orchestra.remote:timed out 2023-12-04T08:49:32.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-04T08:49:32.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T08:49:32.377 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-12-04T08:49:32.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-04T08:49:32.993 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-04T08:49:32.993 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-12-04T08:49:33.008 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-388.el9.x86_64 2023-12-04T08:49:33.008 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-04T08:49:33.009 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T08:49:33.009 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-04T08:49:34.010 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-04T08:49:34.010 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-04T08:49:34.078 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2023-12-04T08:49:34.119 DEBUG:teuthology.parallel:result is None 2023-12-04T08:49:34.120 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T08:49:34.127 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T08:49:34.127 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T08:49:34.142 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T08:49:34.150 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T08:49:34.166 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T08:49:34.167 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T08:49:34.212 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T08:49:34.220 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T08:49:34.220 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T08:49:34.306 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T08:49:34.322 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T08:49:34.324 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T08:49:34.331 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T08:49:34.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T08:49:34.362 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T08:49:34.377 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T08:49:34.378 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T08:49:34.419 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T08:49:34.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T08:49:34.558 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T08:49:34.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T08:49:34.686 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T08:49:34.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T08:49:34.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2023-12-04T08:49:34.848 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T08:49:35.210 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T08:49:35.219 INFO:teuthology.task.internal:Starting timer... 2023-12-04T08:49:35.219 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T08:49:35.230 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T08:49:35.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2023-12-04T08:49:35.305 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2023-12-04T08:49:35.306 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T08:49:35.307 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2023-12-04T08:49:35.362 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2023-12-04T08:49:35.363 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive'} 2023-12-04T08:49:35.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T08:49:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T08:49:35.435 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T08:49:35.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2023-12-04T08:49:35.509 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T08:49:35.522 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-04T08:49:35.522 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T08:49:35.569 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T08:49:35.571 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T08:59:42.392 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com')] 2023-12-04T08:59:42.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-04T08:59:42.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T08:59:42.530 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-12-04T08:59:42.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-12-04T08:59:42.599 INFO:teuthology.run_tasks:Running task clock... 2023-12-04T08:59:42.609 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-04T08:59:42.609 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T08:59:42.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T08:59:42.674 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-04T08:59:42.694 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-04T08:59:42.730 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2023-12-04T08:59:42.748 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2023-12-04T08:59:42.767 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-04T08:59:42.787 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-04T08:59:42.834 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2023-12-04T08:59:42.837 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-04T08:59:42.837 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2023-12-04T08:59:42.837 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T08:59:42.837 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T08:59:42.837 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T08:59:42.837 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T08:59:42.838 INFO:teuthology.run_tasks:Running task install... 2023-12-04T08:59:42.847 DEBUG:teuthology.task.install:project ceph 2023-12-04T08:59:42.848 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607'}} 2023-12-04T08:59:42.848 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607'} 2023-12-04T08:59:42.848 INFO:teuthology.task.install:Using flavor: crimson 2023-12-04T08:59:42.854 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-04T08:59:42.854 INFO:teuthology.task.install:extra packages: [] 2023-12-04T08:59:42.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607', 'tag': None, 'wait_for_package': False} 2023-12-04T08:59:42.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=ebb1a6e3649b44fd77bc9418b26b48a91c567607 2023-12-04T08:59:43.000 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-submit-error-cleanup-testing/ebb1a6e3649b44fd77bc9418b26b48a91c567607/centos/9/flavors/crimson/ 2023-12-04T08:59:43.000 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7693.gebb1a6e3 2023-12-04T08:59:43.429 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-submit-error-cleanup-testing/ebb1a6e3649b44fd77bc9418b26b48a91c567607/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-submit-error-cleanup-testing/ebb1a6e3649b44fd77bc9418b26b48a91c567607/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-submit-error-cleanup-testing/ebb1a6e3649b44fd77bc9418b26b48a91c567607/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-04T08:59:43.429 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T08:59:43.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-04T08:59:43.464 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-04T08:59:43.464 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ebb1a6e3649b44fd77bc9418b26b48a91c567607/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-04T08:59:43.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-04T08:59:43.632 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-04T08:59:43.662 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2023-12-04T08:59:43.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2023-12-04T08:59:43.922 INFO:teuthology.orchestra.run.smithi006.stdout:73 files removed 2023-12-04T08:59:43.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2023-12-04T08:59:45.298 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 80 kB/s | 81 kB 00:01 2023-12-04T08:59:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 10 kB/s | 9.5 kB 00:00 2023-12-04T08:59:47.008 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2023-12-04T08:59:48.519 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 4.4 MB/s | 6.4 MB 00:01 2023-12-04T08:59:50.211 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2023-12-04T08:59:52.726 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 7.9 MB 00:02 2023-12-04T08:59:54.969 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 17 MB/s | 18 MB 00:01 2023-12-04T08:59:59.928 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.8 MB 00:01 2023-12-04T09:00:01.462 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2023-12-04T09:00:04.461 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 20 MB 00:02 2023-12-04T09:00:09.406 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-04T09:00:12.259 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 8.4 M 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 3.1 M 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 3.0 M 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:00:12.263 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 6.6 M 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 17 M 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 24 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 728 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 471 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-04T09:00:12.264 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 4.5 M 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 45 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 129 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 162 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 321 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 298 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 99 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:00:12.265 INFO:teuthology.orchestra.run.smithi006.stdout:Install 27 Packages 2023-12-04T09:00:12.266 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2023-12-04T09:00:12.266 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:12.266 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 54 M 2023-12-04T09:00:12.266 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:00:16.681 INFO:teuthology.orchestra.run.smithi006.stdout:(1/29): ceph-common-18.0.0-7693.gebb1a6e3.el9.x 4.0 MB/s | 17 MB 00:04 2023-12-04T09:00:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:(2/29): ceph-selinux-18.0.0-7693.gebb1a6e3.el9. 290 kB/s | 24 kB 00:00 2023-12-04T09:00:16.940 INFO:teuthology.orchestra.run.smithi006.stdout:(3/29): libcephfs2-18.0.0-7693.gebb1a6e3.el9.x8 4.1 MB/s | 728 kB 00:00 2023-12-04T09:00:17.055 INFO:teuthology.orchestra.run.smithi006.stdout:(4/29): libradosstriper1-18.0.0-7693.gebb1a6e3. 4.0 MB/s | 471 kB 00:00 2023-12-04T09:00:17.789 INFO:teuthology.orchestra.run.smithi006.stdout:(5/29): librgw2-18.0.0-7693.gebb1a6e3.el9.x86_6 6.1 MB/s | 4.5 MB 00:00 2023-12-04T09:00:17.881 INFO:teuthology.orchestra.run.smithi006.stdout:(6/29): python3-ceph-argparse-18.0.0-7693.gebb1 494 kB/s | 45 kB 00:00 2023-12-04T09:00:17.990 INFO:teuthology.orchestra.run.smithi006.stdout:(7/29): python3-ceph-common-18.0.0-7693.gebb1a6 1.2 MB/s | 129 kB 00:00 2023-12-04T09:00:18.107 INFO:teuthology.orchestra.run.smithi006.stdout:(8/29): python3-cephfs-18.0.0-7693.gebb1a6e3.el 1.4 MB/s | 162 kB 00:00 2023-12-04T09:00:18.190 INFO:teuthology.orchestra.run.smithi006.stdout:(9/29): python3-rados-18.0.0-7693.gebb1a6e3.el9 3.8 MB/s | 321 kB 00:00 2023-12-04T09:00:18.274 INFO:teuthology.orchestra.run.smithi006.stdout:(10/29): ceph-base-18.0.0-7693.gebb1a6e3.el9.x8 1.1 MB/s | 6.6 MB 00:05 2023-12-04T09:00:18.299 INFO:teuthology.orchestra.run.smithi006.stdout:(11/29): python3-rbd-18.0.0-7693.gebb1a6e3.el9. 2.7 MB/s | 298 kB 00:00 2023-12-04T09:00:18.391 INFO:teuthology.orchestra.run.smithi006.stdout:(12/29): python3-rgw-18.0.0-7693.gebb1a6e3.el9. 851 kB/s | 99 kB 00:00 2023-12-04T09:00:18.575 INFO:teuthology.orchestra.run.smithi006.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 384 kB/s | 106 kB 00:00 2023-12-04T09:00:18.600 INFO:teuthology.orchestra.run.smithi006.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 769 kB/s | 159 kB 00:00 2023-12-04T09:00:18.634 INFO:teuthology.orchestra.run.smithi006.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 760 kB/s | 44 kB 00:00 2023-12-04T09:00:18.751 INFO:teuthology.orchestra.run.smithi006.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2023-12-04T09:00:18.785 INFO:teuthology.orchestra.run.smithi006.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2023-12-04T09:00:18.818 INFO:teuthology.orchestra.run.smithi006.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 618 kB/s | 41 kB 00:00 2023-12-04T09:00:19.393 INFO:teuthology.orchestra.run.smithi006.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 508 kB/s | 309 kB 00:00 2023-12-04T09:00:19.494 INFO:teuthology.orchestra.run.smithi006.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 243 kB/s | 24 kB 00:00 2023-12-04T09:00:19.586 INFO:teuthology.orchestra.run.smithi006.stdout:(21/29): ceph-radosgw-18.0.0-7693.gebb1a6e3.el9 1.2 MB/s | 8.4 MB 00:07 2023-12-04T09:00:19.603 INFO:teuthology.orchestra.run.smithi006.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2023-12-04T09:00:19.719 INFO:teuthology.orchestra.run.smithi006.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2023-12-04T09:00:19.786 INFO:teuthology.orchestra.run.smithi006.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 4.5 MB/s | 837 kB 00:00 2023-12-04T09:00:19.820 INFO:teuthology.orchestra.run.smithi006.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2023-12-04T09:00:19.979 INFO:teuthology.orchestra.run.smithi006.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2023-12-04T09:00:20.004 INFO:teuthology.orchestra.run.smithi006.stdout:(27/29): libunwind-1.6.2-1.el9.x86_64.rpm 161 kB/s | 67 kB 00:00 2023-12-04T09:00:20.912 INFO:teuthology.orchestra.run.smithi006.stdout:(28/29): librbd1-18.0.0-7693.gebb1a6e3.el9.x86_ 3.2 MB/s | 3.0 MB 00:00 2023-12-04T09:00:21.979 INFO:teuthology.orchestra.run.smithi006.stdout:(29/29): librados2-18.0.0-7693.gebb1a6e3.el9.x8 1.4 MB/s | 3.1 MB 00:02 2023-12-04T09:00:21.982 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:00:21.983 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.5 MB/s | 54 MB 00:09 2023-12-04T09:00:21.983 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:00:22.024 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:00:22.025 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:00:22.520 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:00:22.520 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:00:23.928 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-04T09:00:24.365 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/31 2023-12-04T09:00:24.484 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/31 2023-12-04T09:00:24.574 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:18.0.0-7693.gebb1a6e3.el9.x86_64 3/31 2023-12-04T09:00:24.600 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 4/31 2023-12-04T09:00:24.692 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 4/31 2023-12-04T09:00:24.721 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-12-04T09:00:24.746 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-12-04T09:00:24.776 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-12-04T09:00:24.820 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:18.0.0-7693.gebb1a6e3.el9. 8/31 2023-12-04T09:00:24.889 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:18.0.0-7693.gebb1a6e3.el9.x86_64 9/31 2023-12-04T09:00:24.911 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:18.0.0-7693.gebb1a6e3.el9.x86_6 10/31 2023-12-04T09:00:25.002 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7693.gebb1a6e3.el9.x86_6 10/31 2023-12-04T09:00:25.038 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-12-04T09:00:25.148 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-04T09:00:25.180 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-04T09:00:25.222 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-04T09:00:25.264 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-04T09:00:25.299 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-04T09:00:25.332 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-04T09:00:25.663 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-04T09:00:25.686 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:18.0.0-7693.gebb1a6e3.el9.x86_64 19/31 2023-12-04T09:00:25.785 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:18.0.0-7693.gebb1a6e3.el9.x86_64 19/31 2023-12-04T09:00:25.817 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 20/31 2023-12-04T09:00:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-04T09:00:26.293 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-04T09:00:26.331 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 23/31 2023-12-04T09:00:26.402 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 23/31 2023-12-04T09:00:26.482 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 24/31 2023-12-04T09:00:26.508 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x8 25/31 2023-12-04T09:00:28.645 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x86_64 26/31 2023-12-04T09:00:28.688 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x86_64 26/31 2023-12-04T09:00:29.470 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x86_64 26/31 2023-12-04T09:00:29.558 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:18.0.0-7693.gebb1a6e3.el9.x86_64 27/31 2023-12-04T09:00:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:18.0.0-7693.gebb1a6e3.el9.x86_64 27/31 2023-12-04T09:00:29.674 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-04T09:00:29.674 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-04T09:00:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:29.710 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:18.0.0-7693.gebb1a6e3.el9.x86_64 28/31 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7693.gebb1a6e3.el9.x86_64 28/31 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2023-12-04T09:00:39.570 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2023-12-04T09:00:39.571 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2023-12-04T09:00:39.571 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:40.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 29/31 2023-12-04T09:00:40.471 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 29/31 2023-12-04T09:00:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:00:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-04T09:00:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-04T09:00:40.472 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-04T09:00:40.472 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:40.474 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-04T09:00:40.556 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-04T09:00:40.556 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 3/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:18.0.0-7693.gebb1a6e3.el9.x86_64 4/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 5/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:18.0.0-7693.gebb1a6e3.el9.x86_6 6/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 7/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7693.gebb1a6e3.el9. 8/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x8 9/31 2023-12-04T09:00:41.708 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:18.0.0-7693.gebb1a6e3.el9.x86_64 10/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:18.0.0-7693.gebb1a6e3.el9.x86_64 11/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 12/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 13/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-04T09:00:41.709 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-04T09:00:41.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-04T09:00:41.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 28/31 2023-12-04T09:00:41.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-04T09:00:41.710 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:18.0.0-7693.gebb1a6e3.el9.x86_64 30/31 2023-12-04T09:00:42.806 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-04T09:00:42.806 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.807 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:00:42.808 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2023-12-04T09:00:42.809 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-04T09:00:42.809 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:42.809 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:00:43.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2023-12-04T09:00:43.753 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:34 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:00:43.911 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 117 M 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-04T09:00:43.912 INFO:teuthology.orchestra.run.smithi006.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 158 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout:Install 10 Packages 2023-12-04T09:00:43.913 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:00:43.914 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 120 M 2023-12-04T09:00:43.914 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.1 G 2023-12-04T09:00:43.914 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:00:44.360 INFO:teuthology.orchestra.run.smithi006.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 702 kB/s | 186 kB 00:00 2023-12-04T09:00:44.426 INFO:teuthology.orchestra.run.smithi006.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 6.1 MB/s | 418 kB 00:00 2023-12-04T09:00:44.468 INFO:teuthology.orchestra.run.smithi006.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-12-04T09:00:44.510 INFO:teuthology.orchestra.run.smithi006.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-04T09:00:44.569 INFO:teuthology.orchestra.run.smithi006.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-12-04T09:00:44.594 INFO:teuthology.orchestra.run.smithi006.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2023-12-04T09:00:44.719 INFO:teuthology.orchestra.run.smithi006.stdout:(7/10): libcephsqlite-18.0.0-7693.gebb1a6e3.el9 253 kB/s | 158 kB 00:00 2023-12-04T09:00:44.820 INFO:teuthology.orchestra.run.smithi006.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.2 MB/s | 122 kB 00:00 2023-12-04T09:00:44.853 INFO:teuthology.orchestra.run.smithi006.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 4.9 MB/s | 1.3 MB 00:00 2023-12-04T09:01:30.634 INFO:teuthology.orchestra.run.smithi006.stdout:(10/10): ceph-test-18.0.0-7693.gebb1a6e3.el9.x8 2.5 MB/s | 117 MB 00:46 2023-12-04T09:01:30.637 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:01:30.637 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.6 MB/s | 120 MB 00:46 2023-12-04T09:01:30.637 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:01:30.658 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:01:30.658 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:01:31.513 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:01:31.513 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:01:32.665 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:01:32.756 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-12-04T09:01:32.791 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-12-04T09:01:32.836 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-12-04T09:01:32.875 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-12-04T09:01:32.909 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-12-04T09:01:32.941 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-12-04T09:01:32.969 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-12-04T09:01:33.012 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-12-04T09:01:33.038 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:18.0.0-7693.gebb1a6e3.el9.x86_64 9/10 2023-12-04T09:01:50.263 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7693.gebb1a6e3.el9.x86_64 9/10 2023-12-04T09:01:50.342 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:18.0.0-7693.gebb1a6e3.el9.x86_64 10/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:18.0.0-7693.gebb1a6e3.el9.x86_64 10/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-12-04T09:01:58.638 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-04T09:01:59.096 INFO:teuthology.orchestra.run.smithi006.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-12-04T09:01:59.097 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:01:59.097 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:01:59.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2023-12-04T09:02:00.036 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:51 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:02:00.198 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 6.6 k 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:02:00.200 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 2.0 M 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 1.5 M 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 243 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 8.9 M 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 118 M 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-04T09:02:00.201 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-04T09:02:00.202 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout:Install 36 Packages 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:02:00.203 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 134 M 2023-12-04T09:02:00.204 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.4 G 2023-12-04T09:02:00.204 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:02:01.028 INFO:teuthology.orchestra.run.smithi006.stdout:(1/36): ceph-18.0.0-7693.gebb1a6e3.el9.x86_64.r 24 kB/s | 6.6 kB 00:00 2023-12-04T09:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout:(2/36): ceph-mds-18.0.0-7693.gebb1a6e3.el9.x86_ 1.1 MB/s | 2.0 MB 00:01 2023-12-04T09:02:03.003 INFO:teuthology.orchestra.run.smithi006.stdout:(3/36): ceph-mgr-18.0.0-7693.gebb1a6e3.el9.x86_ 660 kB/s | 1.5 MB 00:02 2023-12-04T09:02:03.220 INFO:teuthology.orchestra.run.smithi006.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7693.gebb1 1.1 MB/s | 243 kB 00:00 2023-12-04T09:02:03.354 INFO:teuthology.orchestra.run.smithi006.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-12-04T09:02:03.404 INFO:teuthology.orchestra.run.smithi006.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2023-12-04T09:02:03.446 INFO:teuthology.orchestra.run.smithi006.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2023-12-04T09:02:03.487 INFO:teuthology.orchestra.run.smithi006.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2023-12-04T09:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 927 kB/s | 39 kB 00:00 2023-12-04T09:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-04T09:02:03.647 INFO:teuthology.orchestra.run.smithi006.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-04T09:02:03.680 INFO:teuthology.orchestra.run.smithi006.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-04T09:02:03.714 INFO:teuthology.orchestra.run.smithi006.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2023-12-04T09:02:03.748 INFO:teuthology.orchestra.run.smithi006.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-04T09:02:03.781 INFO:teuthology.orchestra.run.smithi006.stdout:(15/36): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-04T09:02:03.907 INFO:teuthology.orchestra.run.smithi006.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2023-12-04T09:02:04.207 INFO:teuthology.orchestra.run.smithi006.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 504 kB/s | 151 kB 00:00 2023-12-04T09:02:04.324 INFO:teuthology.orchestra.run.smithi006.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2023-12-04T09:02:04.482 INFO:teuthology.orchestra.run.smithi006.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2023-12-04T09:02:04.616 INFO:teuthology.orchestra.run.smithi006.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2023-12-04T09:02:04.741 INFO:teuthology.orchestra.run.smithi006.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 86 kB/s | 11 kB 00:00 2023-12-04T09:02:04.850 INFO:teuthology.orchestra.run.smithi006.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2023-12-04T09:02:04.967 INFO:teuthology.orchestra.run.smithi006.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2023-12-04T09:02:05.109 INFO:teuthology.orchestra.run.smithi006.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 137 kB/s | 19 kB 00:00 2023-12-04T09:02:05.192 INFO:teuthology.orchestra.run.smithi006.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2023-12-04T09:02:05.268 INFO:teuthology.orchestra.run.smithi006.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 618 kB/s | 46 kB 00:00 2023-12-04T09:02:05.343 INFO:teuthology.orchestra.run.smithi006.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-04T09:02:05.418 INFO:teuthology.orchestra.run.smithi006.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2023-12-04T09:02:05.493 INFO:teuthology.orchestra.run.smithi006.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2023-12-04T09:02:05.577 INFO:teuthology.orchestra.run.smithi006.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2023-12-04T09:02:05.669 INFO:teuthology.orchestra.run.smithi006.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 391 kB/s | 36 kB 00:00 2023-12-04T09:02:05.753 INFO:teuthology.orchestra.run.smithi006.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-12-04T09:02:05.845 INFO:teuthology.orchestra.run.smithi006.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.6 MB/s | 427 kB 00:00 2023-12-04T09:02:05.920 INFO:teuthology.orchestra.run.smithi006.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 267 kB/s | 20 kB 00:00 2023-12-04T09:02:07.503 INFO:teuthology.orchestra.run.smithi006.stdout:(35/36): ceph-mon-18.0.0-7693.gebb1a6e3.el9.x86 1.4 MB/s | 8.9 MB 00:06 2023-12-04T09:04:12.025 INFO:teuthology.orchestra.run.smithi006.stdout:(36/36): ceph-osd-18.0.0-7693.gebb1a6e3.el9.x86 930 kB/s | 118 MB 02:09 2023-12-04T09:04:12.027 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:04:12.027 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.0 MB/s | 134 MB 02:11 2023-12-04T09:04:12.038 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:04:12.080 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:04:12.080 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:04:13.154 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:04:13.154 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:04:15.137 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:04:15.201 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-04T09:04:15.244 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-04T09:04:15.279 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-04T09:04:15.353 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-04T09:04:15.404 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-04T09:04:15.436 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-04T09:04:15.532 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-04T09:04:15.609 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-04T09:04:15.665 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-04T09:04:15.737 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-04T09:04:15.790 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-04T09:04:15.828 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-04T09:04:15.886 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-04T09:04:15.950 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-04T09:04:16.148 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-04T09:04:16.199 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-04T09:04:16.238 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-04T09:04:16.272 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-04T09:04:16.302 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-04T09:04:16.381 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-04T09:04:16.421 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-04T09:04:16.454 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-04T09:04:16.495 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-04T09:04:16.538 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-04T09:04:16.572 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-04T09:04:16.664 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-04T09:04:16.704 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-04T09:04:16.779 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-04T09:04:16.928 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-04T09:04:17.059 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-04T09:04:17.231 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7693.gebb1a6e3.el9. 31/36 2023-12-04T09:04:17.274 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:18.0.0-7693.gebb1a6e3.el9.x86_64 32/36 2023-12-04T09:04:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7693.gebb1a6e3.el9.x86_64 32/36 2023-12-04T09:04:17.301 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:04:17.301 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-04T09:04:17.301 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-04T09:04:17.301 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-04T09:04:17.301 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:36.615 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 33/36 2023-12-04T09:04:36.643 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 33/36 2023-12-04T09:04:36.643 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:04:36.643 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-04T09:04:36.643 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-04T09:04:36.643 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-04T09:04:36.643 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:37.574 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:18.0.0-7693.gebb1a6e3.el9.x86_64 34/36 2023-12-04T09:04:37.598 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:18.0.0-7693.gebb1a6e3.el9.x86_64 34/36 2023-12-04T09:04:37.598 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:04:37.598 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-04T09:04:37.598 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-04T09:04:37.598 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-04T09:04:37.598 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:37.808 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:18.0.0-7693.gebb1a6e3.el9.x86_64 35/36 2023-12-04T09:04:37.832 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:18.0.0-7693.gebb1a6e3.el9.x86_64 35/36 2023-12-04T09:04:37.832 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:04:37.832 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-04T09:04:37.832 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-04T09:04:37.832 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-04T09:04:37.832 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:37.900 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:18.0.0-7693.gebb1a6e3.el9.x86_64 36/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:18.0.0-7693.gebb1a6e3.el9.x86_64 36/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:18.0.0-7693.gebb1a6e3.el9.x86_64 3/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:18.0.0-7693.gebb1a6e3.el9.x86_64 4/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 5/36 2023-12-04T09:04:47.955 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7693.gebb1a6e3.el9. 6/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-04T09:04:47.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-04T09:04:47.957 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:04:48.960 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-04T09:04:48.961 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-04T09:04:48.962 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-04T09:04:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:48.963 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:04:49.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2023-12-04T09:04:49.814 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:40 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:04:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:04:49.967 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:04:49.968 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:04:49.968 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:04:50.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2023-12-04T09:04:50.635 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:41 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:04:50.785 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:04:50.786 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:50.786 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:04:50.786 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:50.786 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:04:50.786 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 776 k 2023-12-04T09:04:50.786 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 776 k 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 790 k 2023-12-04T09:04:50.787 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:04:52.415 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-18.0.0-7693.gebb1a6e3.el9.noarch.rpm 477 kB/s | 776 kB 00:01 2023-12-04T09:04:52.415 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:04:52.416 INFO:teuthology.orchestra.run.smithi006.stdout:Total 476 kB/s | 776 kB 00:01 2023-12-04T09:04:52.416 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:04:52.423 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:04:52.423 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:04:52.429 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:04:52.430 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:04:52.504 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:04:52.995 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:04:53.114 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:04:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:04:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:04:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:53.842 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:04:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:04:53.842 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:53.842 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:04:54.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2023-12-04T09:04:54.657 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:45 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:04:54.807 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 143 k 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:04:54.808 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:54.809 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 143 k 2023-12-04T09:04:54.809 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 438 k 2023-12-04T09:04:54.809 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:04:55.587 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-18.0.0-7693.gebb1a6 184 kB/s | 143 kB 00:00 2023-12-04T09:04:55.587 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:04:55.588 INFO:teuthology.orchestra.run.smithi006.stdout:Total 184 kB/s | 143 kB 00:00 2023-12-04T09:04:55.588 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:04:55.597 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:04:55.597 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:04:55.655 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:04:55.656 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:04:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:04:55.888 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7693.gebb1a6e3. 1/1 2023-12-04T09:04:55.913 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7693.gebb1a6e3. 1/1 2023-12-04T09:04:55.913 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:04:55.914 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-04T09:04:55.914 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:56.965 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7693.gebb1a6e3. 1/1 2023-12-04T09:04:56.965 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:56.965 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:04:56.965 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:04:56.965 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:56.965 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:04:57.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2023-12-04T09:04:57.802 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:48 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:04:57.900 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:04:57.953 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:04:57.954 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:04:57.954 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:04:58.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2023-12-04T09:04:58.620 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:49 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:04:58.772 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:04:58.773 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:58.773 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 1.8 M 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 28 k 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 15 k 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:58.774 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:04:58.775 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:04:58.775 INFO:teuthology.orchestra.run.smithi006.stdout:Install 7 Packages 2023-12-04T09:04:58.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:04:58.775 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.1 M 2023-12-04T09:04:58.775 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 8.7 M 2023-12-04T09:04:58.775 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:04:59.565 INFO:teuthology.orchestra.run.smithi006.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7693.gebb 65 kB/s | 28 kB 00:00 2023-12-04T09:04:59.774 INFO:teuthology.orchestra.run.smithi006.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 43 kB/s | 9.0 kB 00:00 2023-12-04T09:04:59.924 INFO:teuthology.orchestra.run.smithi006.stdout:(3/7): python3-jwt-2.4.0-1.el9.noarch.rpm 272 kB/s | 41 kB 00:00 2023-12-04T09:05:00.016 INFO:teuthology.orchestra.run.smithi006.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 335 kB/s | 31 kB 00:00 2023-12-04T09:05:00.191 INFO:teuthology.orchestra.run.smithi006.stdout:(5/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2023-12-04T09:05:01.708 INFO:teuthology.orchestra.run.smithi006.stdout:(6/7): ceph-prometheus-alerts-18.0.0-7693.gebb1 5.9 kB/s | 15 kB 00:02 2023-12-04T09:05:02.092 INFO:teuthology.orchestra.run.smithi006.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7693.gebb1a6e3 609 kB/s | 1.8 MB 00:02 2023-12-04T09:05:02.096 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:02.096 INFO:teuthology.orchestra.run.smithi006.stdout:Total 636 kB/s | 2.1 MB 00:03 2023-12-04T09:05:02.096 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:02.106 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:02.106 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:02.185 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:02.187 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:02.693 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-04T09:05:02.728 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-04T09:05:02.775 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-04T09:05:02.805 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-04T09:05:02.836 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7693.gebb1a6e3.el9.n 5/7 2023-12-04T09:05:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7693.gebb1a6e3.el9. 6/7 2023-12-04T09:05:03.112 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7693.gebb1a6e3.el9.noarc 7/7 2023-12-04T09:05:03.621 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7693.gebb1a6e3.el9.noarc 7/7 2023-12-04T09:05:03.621 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7693.gebb1a6e3.el9. 1/7 2023-12-04T09:05:03.621 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7693.gebb1a6e3.el9.noarc 2/7 2023-12-04T09:05:03.621 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7693.gebb1a6e3.el9.n 3/7 2023-12-04T09:05:03.621 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-04T09:05:03.622 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-04T09:05:03.622 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:03.956 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:04.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-04T09:05:04.783 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:04:55 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:04.935 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:04.936 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2023-12-04T09:05:04.936 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:05:04.936 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2023-12-04T09:05:04.936 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:04.936 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 7.4 M 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:04.937 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2023-12-04T09:05:04.938 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2023-12-04T09:05:04.938 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:04.938 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 37 M 2023-12-04T09:05:04.938 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 209 M 2023-12-04T09:05:04.938 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:05.115 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 297 kB/s | 32 kB 00:00 2023-12-04T09:05:05.199 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 287 kB/s | 17 kB 00:00 2023-12-04T09:05:05.232 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-04T09:05:05.424 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-12-04T09:05:05.649 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2023-12-04T09:05:05.691 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2023-12-04T09:05:05.950 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2023-12-04T09:05:06.233 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2023-12-04T09:05:06.267 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2023-12-04T09:05:06.292 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-12-04T09:05:07.134 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 283 kB/s | 245 kB 00:00 2023-12-04T09:05:17.617 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 601 kB/s | 7.4 MB 00:12 2023-12-04T09:05:17.620 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:17.620 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.9 MB/s | 37 MB 00:12 2023-12-04T09:05:17.634 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:17.661 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:17.661 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:18.063 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:18.063 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:18.790 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:18.863 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-04T09:05:18.896 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-04T09:05:18.995 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-04T09:05:19.041 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-04T09:05:19.226 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-04T09:05:19.290 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-04T09:05:19.378 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-04T09:05:20.068 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-04T09:05:20.182 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-04T09:05:21.516 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-04T09:05:22.595 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-04T09:05:22.622 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7693.gebb1a 12/12 2023-12-04T09:05:25.039 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7693.gebb1a 12/12 2023-12-04T09:05:25.039 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7693.gebb1a 1/12 2023-12-04T09:05:25.039 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-04T09:05:25.040 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-04T09:05:25.457 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:25.458 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:25.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2023-12-04T09:05:26.422 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:17 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:26.576 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:26.577 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 50 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-04T09:05:26.578 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:26.579 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2023-12-04T09:05:26.580 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2023-12-04T09:05:26.580 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:26.773 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 220 kB/s | 24 kB 00:00 2023-12-04T09:05:26.823 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2023-12-04T09:05:26.873 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2023-12-04T09:05:26.899 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-12-04T09:05:26.924 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-04T09:05:26.949 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 879 kB/s | 43 kB 00:00 2023-12-04T09:05:27.033 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 180 kB/s | 15 kB 00:00 2023-12-04T09:05:27.058 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 241 kB/s | 32 kB 00:00 2023-12-04T09:05:27.150 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-google-auth-2.23.4-1.el9.noarch 1.6 MB/s | 197 kB 00:00 2023-12-04T09:05:27.201 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2023-12-04T09:05:27.242 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2023-12-04T09:05:27.267 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2023-12-04T09:05:27.309 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): ceph-mgr-rook-18.0.0-7693.gebb1a6e3.el 78 kB/s | 50 kB 00:00 2023-12-04T09:05:27.311 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:27.312 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2023-12-04T09:05:27.312 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:27.327 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:27.327 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:27.442 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:27.444 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:27.984 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:28.165 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-04T09:05:28.220 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-04T09:05:28.276 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-04T09:05:28.312 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-04T09:05:28.354 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-04T09:05:28.451 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-04T09:05:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-04T09:05:28.614 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-04T09:05:29.374 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-04T09:05:29.435 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-04T09:05:29.469 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-04T09:05:29.517 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-04T09:05:29.546 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:18.0.0-7693.gebb1a6e3.el9.noarch 13/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7693.gebb1a6e3.el9.noarch 13/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-04T09:05:30.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-04T09:05:30.689 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-04T09:05:30.690 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-04T09:05:30.690 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-04T09:05:30.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:30.690 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:30.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2023-12-04T09:05:31.540 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:22 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:31.694 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 133 k 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-04T09:05:31.695 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:31.696 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.8 M 2023-12-04T09:05:31.697 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2023-12-04T09:05:31.697 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:31.927 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-04T09:05:32.161 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2023-12-04T09:05:32.552 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7693.gebb1a6e3.e 171 kB/s | 133 kB 00:00 2023-12-04T09:05:32.594 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 133 kB/s | 58 kB 00:00 2023-12-04T09:05:32.661 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 747 kB/s | 548 kB 00:00 2023-12-04T09:05:33.028 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 107 kB/s | 51 kB 00:00 2023-12-04T09:05:33.033 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:33.033 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2023-12-04T09:05:33.034 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:33.044 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:33.044 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:33.143 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:33.143 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:33.475 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:33.662 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-04T09:05:33.716 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-04T09:05:34.271 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-04T09:05:34.347 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-04T09:05:34.423 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-04T09:05:34.452 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 6/6 2023-12-04T09:05:35.273 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 6/6 2023-12-04T09:05:35.273 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/6 2023-12-04T09:05:35.273 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-04T09:05:35.273 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-04T09:05:35.273 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-04T09:05:35.273 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-04T09:05:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:35.598 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:35.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2023-12-04T09:05:36.481 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:27 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:36.632 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 833 k 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:05:36.633 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 912 k 2023-12-04T09:05:36.634 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 3.0 M 2023-12-04T09:05:36.635 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:36.749 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 79 kB 00:00 2023-12-04T09:05:37.890 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-fuse-18.0.0-7693.gebb1a6e3.el9.x86_ 663 kB/s | 833 kB 00:01 2023-12-04T09:05:37.892 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:37.892 INFO:teuthology.orchestra.run.smithi006.stdout:Total 724 kB/s | 912 kB 00:01 2023-12-04T09:05:37.893 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:37.918 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:37.919 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:38.004 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:38.004 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:38.205 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:38.327 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-04T09:05:38.392 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/2 2023-12-04T09:05:39.132 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/2 2023-12-04T09:05:39.132 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/2 2023-12-04T09:05:39.473 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-04T09:05:39.473 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:39.473 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:39.473 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-04T09:05:39.473 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:39.474 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:39.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2023-12-04T09:05:40.346 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:31 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:40.498 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:18.0.0-7693.gebb1a6e3.el9 ceph-noarch 262 k 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:05:40.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:40.500 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 262 k 2023-12-04T09:05:40.500 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.2 M 2023-12-04T09:05:40.500 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:41.228 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-volume-18.0.0-7693.gebb1a6e3.el9.noarch.rp 361 kB/s | 262 kB 00:00 2023-12-04T09:05:41.228 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:41.229 INFO:teuthology.orchestra.run.smithi006.stdout:Total 360 kB/s | 262 kB 00:00 2023-12-04T09:05:41.229 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:41.238 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:41.238 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:41.274 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:41.274 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:41.496 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:41.571 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:05:41.588 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:05:41.588 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:05:41.588 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-04T09:05:41.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:42.714 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:18.0.0-7693.gebb1a6e3.el9.noarch 1/1 2023-12-04T09:05:42.714 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:42.714 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:42.714 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:18.0.0-7693.gebb1a6e3.el9.noarch 2023-12-04T09:05:42.714 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:42.714 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:43.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2023-12-04T09:05:43.668 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:34 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:43.821 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:43.821 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:43.821 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-04T09:05:43.821 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:43.821 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:43.821 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 127 k 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 127 k 2023-12-04T09:05:43.822 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 460 k 2023-12-04T09:05:43.823 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:44.508 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-18.0.0-7693.gebb1a6e3.el9.x86_64 185 kB/s | 127 kB 00:00 2023-12-04T09:05:44.508 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:44.509 INFO:teuthology.orchestra.run.smithi006.stdout:Total 185 kB/s | 127 kB 00:00 2023-12-04T09:05:44.509 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:44.516 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:44.516 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:44.577 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:44.577 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:44.729 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:44.834 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:45.264 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:45.593 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:45.593 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:45.593 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:45.594 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:05:45.594 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:45.594 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:45.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2023-12-04T09:05:46.479 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:37 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:46.578 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:46.631 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:46.633 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:46.633 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:46.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2023-12-04T09:05:47.310 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:38 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:47.464 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:47.465 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:47.465 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-04T09:05:47.465 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:47.465 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:47.465 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 31 k 2023-12-04T09:05:47.465 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 31 k 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 139 k 2023-12-04T09:05:47.466 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:47.928 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-18.0.0-7693.gebb1a6e3.el9.x86_6 67 kB/s | 31 kB 00:00 2023-12-04T09:05:47.928 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:47.928 INFO:teuthology.orchestra.run.smithi006.stdout:Total 67 kB/s | 31 kB 00:00 2023-12-04T09:05:47.929 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:47.932 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:47.932 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:47.964 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:47.964 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:48.057 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:48.586 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:48.876 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:48.876 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:48.876 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:48.877 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:05:48.877 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:48.877 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:49.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2023-12-04T09:05:49.658 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:40 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:49.757 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:49.811 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:49.812 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:49.812 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:49.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2023-12-04T09:05:50.478 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:41 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:50.577 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:50.631 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:50.632 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:50.632 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:50.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2023-12-04T09:05:51.316 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:42 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:51.415 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:51.471 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:51.472 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:51.472 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:51.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2023-12-04T09:05:52.157 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:43 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:52.257 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:52.311 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:52.312 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:52.312 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:52.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2023-12-04T09:05:52.984 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:43 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:53.083 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:53.137 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:53.139 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:53.139 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:53.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2023-12-04T09:05:53.839 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:44 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:53.938 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 is already installed. 2023-12-04T09:05:53.992 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:53.992 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-12-04T09:05:53.993 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:54.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2023-12-04T09:05:54.664 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:45 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:54.816 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:54.816 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:54.816 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 86 k 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:54.817 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 86 k 2023-12-04T09:05:54.818 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 256 k 2023-12-04T09:05:54.818 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:05:55.948 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-18.0.0-7693.gebb1a6e3.el9.x86_64.rpm 76 kB/s | 86 kB 00:01 2023-12-04T09:05:55.949 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:05:55.949 INFO:teuthology.orchestra.run.smithi006.stdout:Total 76 kB/s | 86 kB 00:01 2023-12-04T09:05:55.949 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:05:55.958 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:05:55.959 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:05:56.021 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:05:56.021 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:05:56.173 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:05:56.282 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:56.702 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:57.006 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:05:57.006 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:57.006 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:05:57.006 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:05:57.006 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:57.006 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:05:57.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2023-12-04T09:05:57.854 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:48 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:05:58.007 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:05:58.007 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:58.007 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-12-04T09:05:58.007 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 3.0 M 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.0 M 2023-12-04T09:05:58.008 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 13 M 2023-12-04T09:05:58.009 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:06:00.843 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-18.0.0-7693.gebb1a6e3.el9.x86_64.rpm 1.1 MB/s | 3.0 MB 00:02 2023-12-04T09:06:00.844 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:06:00.844 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.1 MB/s | 3.0 MB 00:02 2023-12-04T09:06:00.844 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:06:00.856 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:06:00.856 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:06:00.936 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:06:00.936 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:06:01.383 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:06:01.443 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:06:01.470 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:06:01.470 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-04T09:06:01.470 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-04T09:06:01.470 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-04T09:06:01.470 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-04T09:06:01.470 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:06:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:02.581 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:06:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:06:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:02.582 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:06:02.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2023-12-04T09:06:03.434 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:05:54 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:06:03.587 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:06:03.587 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:06:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-12-04T09:06:03.587 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 169 k 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 169 k 2023-12-04T09:06:03.588 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 537 k 2023-12-04T09:06:03.589 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:06:04.254 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-18.0.0-7693.gebb1a6e3.el9.x86_64.rpm 255 kB/s | 169 kB 00:00 2023-12-04T09:06:04.255 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:06:04.255 INFO:teuthology.orchestra.run.smithi006.stdout:Total 254 kB/s | 169 kB 00:00 2023-12-04T09:06:04.255 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:06:04.264 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:06:04.264 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:06:04.322 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:06:04.323 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:06:04.493 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:06:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:06:05.023 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:06:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/1 2023-12-04T09:06:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:05.315 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:06:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:06:05.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:05.316 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:06:05.557 DEBUG:teuthology.parallel:result is None 2023-12-04T09:06:05.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=ebb1a6e3649b44fd77bc9418b26b48a91c567607 2023-12-04T09:06:05.699 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-04T09:06:05.719 INFO:teuthology.orchestra.run.smithi006.stdout:18.0.0-7693.gebb1a6e3.el9 2023-12-04T09:06:05.720 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7693.gebb1a6e3.el9 2023-12-04T09:06:05.720 INFO:teuthology.task.install:The correct ceph version 18.0.0-7693.gebb1a6e3 is installed. 2023-12-04T09:06:05.720 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-04T09:06:05.720 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:05.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T09:06:05.791 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-04T09:06:05.792 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:05.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T09:06:05.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T09:06:05.934 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-04T09:06:05.935 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:05.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T09:06:06.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T09:06:06.075 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-04T09:06:06.075 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:06.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T09:06:06.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T09:06:06.217 INFO:teuthology.run_tasks:Running task ceph... 2023-12-04T09:06:10.476 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-04T09:06:10.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2023-12-04T09:06:10.512 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-04T09:06:10.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T09:06:10.588 INFO:tasks.ceph:Creating extra log directories... 2023-12-04T09:06:10.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T09:06:10.668 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-04T09:06:10.668 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-04T09:06:10.668 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785', 'branch': 'wip-matanb-crimson-submit-error-cleanup-testing', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read}', 'email': None, 'first_in_suite': False, 'job_id': '7476785', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-submit-error-cleanup-testing'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'e89bb53732569ae4a8d78945935b6e5cb35576be'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9400, 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_e89bb53732569ae4a8d78945935b6e5cb35576be/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'e89bb53732569ae4a8d78945935b6e5cb35576be', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJkY88tiDoPV86zWCdQX5Liqvop9xun1VX6pXHPUmWPRNToKKXsI8rrVjM1HldcpZVjNXGnqlF6ItHv0oahRovM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'ebb1a6e3649b44fd77bc9418b26b48a91c567607', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-04_08:39:13', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-04T09:06:10.669 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-04T09:06:10.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T09:06:10.792 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:10.792 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T09:06:10.850 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T09:06:10.850 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2023-12-04T09:06:10.905 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-04 09:06:01.704509166 +0000 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-04 08:59:12.348268694 +0000 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-04 08:59:12.348268694 +0000 2023-12-04T09:06:10.906 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-04 08:59:12.348268694 +0000 2023-12-04T09:06:10.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T09:06:10.977 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-04T09:06:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-04T09:06:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000225608 s, 2.3 MB/s 2023-12-04T09:06:10.979 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T09:06:11.037 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-04 09:06:01.704509166 +0000 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-04 08:59:12.867263318 +0000 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-04 08:59:12.867263318 +0000 2023-12-04T09:06:11.094 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-04 08:59:12.867263318 +0000 2023-12-04T09:06:11.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T09:06:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-04T09:06:11.164 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-04T09:06:11.165 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000189566 s, 2.7 MB/s 2023-12-04T09:06:11.166 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T09:06:11.222 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2023-12-04T09:06:11.278 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-04 09:06:01.705509155 +0000 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-04 08:59:13.387257933 +0000 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-04 08:59:13.387257933 +0000 2023-12-04T09:06:11.279 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-04 08:59:13.387257933 +0000 2023-12-04T09:06:11.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T09:06:11.348 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-04T09:06:11.348 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-04T09:06:11.348 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000180804 s, 2.8 MB/s 2023-12-04T09:06:11.349 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T09:06:11.405 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-12-04 09:06:01.705509155 +0000 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-12-04 08:59:13.927252340 +0000 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-12-04 08:59:13.927252340 +0000 2023-12-04T09:06:11.461 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2023-12-04 08:59:13.927252340 +0000 2023-12-04T09:06:11.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T09:06:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-12-04T09:06:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-12-04T09:06:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000181708 s, 2.8 MB/s 2023-12-04T09:06:11.532 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T09:06:11.588 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T09:06:11.589 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-04T09:06:11.589 INFO:tasks.ceph:Generating config... 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-12-04T09:06:11.590 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] err to stderr = True 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] log flush on exit = True 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] log to file = False 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] log to stderr = True 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-04T09:06:11.591 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-04T09:06:11.591 INFO:tasks.ceph:[global] auth client required = none 2023-12-04T09:06:11.591 INFO:tasks.ceph:[global] auth cluster required = none 2023-12-04T09:06:11.591 INFO:tasks.ceph:[global] auth service required = none 2023-12-04T09:06:11.591 INFO:tasks.ceph:[global] auth supported = none 2023-12-04T09:06:11.591 INFO:tasks.ceph:[global] debug asok = 0/0 2023-12-04T09:06:11.591 INFO:tasks.ceph:[global] debug auth = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug client = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug context = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug crush = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug filer = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-12-04T09:06:11.592 INFO:tasks.ceph:[global] debug journal = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mds = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug mon = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug monc = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug ms = 0/0 2023-12-04T09:06:11.593 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug osd = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rados = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug refs = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug striper = 0/0 2023-12-04T09:06:11.594 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] debug timer = 0/0 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] debug tp = 0/0 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] ms client mode = crc 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-12-04T09:06:11.595 INFO:tasks.ceph:[global] ms service mode = crc 2023-12-04T09:06:11.595 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-04T09:06:11.595 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-04T09:06:11.595 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-12-04T09:06:11.595 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-12-04T09:06:11.595 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-12-04T09:06:11.595 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-12-04T09:06:11.596 INFO:tasks.ceph:Setting up mon.a... 2023-12-04T09:06:11.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-04T09:06:11.691 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2023-12-04T09:06:11.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-04T09:06:11.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T09:06:11.776 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6')] 2023-12-04T09:06:11.777 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-12-04T09:06:11.778 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:11.778 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T09:06:11.832 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:11.915+0000 7fca8a644e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:11.915+0000 7fca8a644e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:11.915+0000 7fca8a644e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:11.917 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:11.917 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 1a67ff37-3baa-4e2c-8b80-4cd582792f60 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = pacific 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 1a67ff37-3baa-4e2c-8b80-4cd582792f60 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2023-12-04T09:06:11.916639+0000 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:created 2023-12-04T09:06:11.916639+0000 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 16 (pacific) 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-12-04T09:06:11.918 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-04T09:06:11.919 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T09:06:11.934 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1a67ff37-3baa-4e2c-8b80-4cd582792f60... 2023-12-04T09:06:11.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = none 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.035 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2023-12-04T09:06:12.036 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.6 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: auth client required = none 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: auth cluster required = none 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: auth service required = none 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: debug asok = 0/0 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 0/0 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: debug bdev = 0/0 2023-12-04T09:06:12.037 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug buffer = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug civetweb = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug compressor = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug context = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug crush = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug crypto = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug dpdk = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug eventtrace = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug filer = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug filestore = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug finisher = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug fuse = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug heartbeatmap = 0/0 2023-12-04T09:06:12.038 INFO:teuthology.orchestra.run.smithi006.stdout: debug javaclient = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug journal = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug journaler = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug kstore = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug leveldb = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug lockdep = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds locker = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds log = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds log expire = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds migrator = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug memdb = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgrc = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 0/0 2023-12-04T09:06:12.039 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug objclass = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug objectcacher = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug objecter = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug optracker = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug perfcounter = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug rados = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd mirror = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd replay = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug refs = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug rgw = 0/0 2023-12-04T09:06:12.040 INFO:teuthology.orchestra.run.smithi006.stdout: debug rgw sync = 0/0 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 0/0 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: debug striper = 0/0 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: debug throttle = 0/0 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: debug timer = 0/0 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: debug tp = 0/0 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: ms client mode = crc 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: ms cluster mode = crc 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: ms mon client mode = crc 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: ms mon cluster mode = crc 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: ms mon service mode = crc 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: ms service mode = crc 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 1a67ff37-3baa-4e2c-8b80-4cd582792f60 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.041 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2023-12-04T09:06:12.042 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2023-12-04T09:06:12.043 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = snappy 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore rocksdb cf = False 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: crimson seastar smp = 3 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug filestore = 0/0 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug journal = 0/0 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 0/0 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 0/0 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: err to stderr = True 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: log flush on exit = True 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: log to file = False 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = True 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T09:06:12.044 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: osd shutdown pgref assert = True 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 0/0 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 0/0 2023-12-04T09:06:12.045 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 0/0 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T09:06:12.046 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default crimson = True 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T09:06:12.047 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2023-12-04T09:06:12.052 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-04T09:06:12.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-04T09:06:12.117 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-04T09:06:12.118 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:12.118 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-04T09:06:12.133 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:12.133 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-04T09:06:12.187 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:06:12.187 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:12.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-04T09:06:12.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T09:06:12.273 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:12.273 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:12.329 INFO:tasks.ceph:Setting up mon nodes... 2023-12-04T09:06:12.329 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-04T09:06:12.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-04T09:06:12.437 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-04T09:06:12.439 INFO:tasks.ceph:Setting up mds nodes... 2023-12-04T09:06:12.439 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-04T09:06:12.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-04T09:06:12.491 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-04T09:06:12.509 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-04T09:06:12.510 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-04T09:06:12.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-04T09:06:12.540 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T09:06:12.540 INFO:tasks.ceph:role: osd.0 2023-12-04T09:06:12.540 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:06:12.540 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-12-04T09:06:12.618 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T09:06:12.619 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T09:06:12.619 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T09:06:12.619 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T09:06:13.125 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-12-04T09:06:13.128 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-12-04T09:06:13.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-04T09:06:13.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-04T09:06:13.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-04T09:06:13.392 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T09:06:13.392 INFO:tasks.ceph:role: osd.1 2023-12-04T09:06:13.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:06:13.392 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T09:06:13.471 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T09:06:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T09:06:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T09:06:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T09:06:13.472 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T09:06:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-12-04T09:06:13.472 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T09:06:13.472 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T09:06:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T09:06:13.472 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T09:06:13.911 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-12-04T09:06:13.913 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-12-04T09:06:13.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-04T09:06:14.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-04T09:06:14.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-04T09:06:14.125 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-04T09:06:14.125 INFO:tasks.ceph:role: osd.2 2023-12-04T09:06:14.125 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:06:14.125 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T09:06:14.204 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T09:06:14.204 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T09:06:14.204 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T09:06:14.204 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-04T09:06:14.204 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T09:06:14.204 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-12-04T09:06:14.205 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T09:06:14.205 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-04T09:06:14.205 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T09:06:14.205 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T09:06:14.696 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-12-04T09:06:14.698 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-12-04T09:06:14.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-04T09:06:14.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-04T09:06:14.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-04T09:06:15.420 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:15,419 seastar - Reactor backend: linux-aio 2023-12-04T09:06:15.436 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:15,435 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-04T09:06:15.436 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:15,435 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-04T09:06:15.436 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:15,435 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-04T09:06:15.436 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:15,435 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-04T09:06:15.438 INFO:teuthology.orchestra.run.smithi006.stderr:==80978==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-04T09:06:15.445 INFO:teuthology.orchestra.run.smithi006.stderr:==80978==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffed7bda000; bottom 0x7feca4724000; size: 0x0012334b6000 (78169989120) 2023-12-04T09:06:15.445 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-04T09:06:15.445 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-04T09:06:15.988 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:15.988 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-04T09:06:15.988 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-04T09:06:15.988 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-04T09:06:15.988 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:15.988 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-04T09:06:16.044 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:16,043 seastar - Reactor backend: linux-aio 2023-12-04T09:06:16.062 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:16,061 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-04T09:06:16.062 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:16,061 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-04T09:06:16.062 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:16,061 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-04T09:06:16.069 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:16,068 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-04T09:06:16.072 INFO:teuthology.orchestra.run.smithi006.stderr:==80976==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-04T09:06:16.078 INFO:teuthology.orchestra.run.smithi006.stderr:==80976==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffed7bda000; bottom 0x7feca251d000; size: 0x0012356bd000 (78205669376) 2023-12-04T09:06:16.078 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-04T09:06:16.078 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-04T09:06:16.104 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c922c6c 0x2c92401a 0x2c925568 0x2c9258ec 0x2c925a36 0x2c925e8c 0x54daf 0xb92c7 0x94469 0x94d46 0x22c42525 0x22c4284d 0x22cceeeb 0x22cd20fb 0x22d5d2e5 0x22d5fe2f 0x22d654a0 0x24c4328b 0x24c49f76 0x24c4c8c4 0x24c4da9d 0x227a99d9 0x227fa636 0x2281e36b 0x2c8c4f87 0x2c962d94 0x2cb872cd 0x2cb89117 0x2c5de54e 0x2c5e0457 0x1ed3586a 0x3feaf 0x3ff5f 0x1e7dea44 2023-12-04T09:06:16.104 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-04T09:06:16.165 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c922c6c 0x2c92401a 0x2c925568 0x2c9258ec 0x2c925a36 0x2c925e8c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x264a04ae 0x264a2af7 0x264a4908 0x264a716d 0x262d8055 0x1edf1f22 0x1ed63adf 0x1ed67920 0x2cca6e49 2023-12-04T09:06:16.165 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff857508a3 0xffffffff857508ea 0xffffffff8626646c 0xffffffff864000ea 2023-12-04T09:06:20.113 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:20.113 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-04T09:06:20.113 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-04T09:06:20.113 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-04T09:06:20.113 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:20.114 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:20.114 INFO:teuthology.orchestra.run.smithi006.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 1a67ff37-3baa-4e2c-8b80-4cd582792f60 2023-12-04T09:06:20.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-04T09:06:20.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:20.730 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-04T09:06:20.731 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:20,730 seastar - Reactor backend: linux-aio 2023-12-04T09:06:20.748 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:20,747 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-04T09:06:20.748 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:20,747 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-04T09:06:20.748 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:20,747 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-04T09:06:20.748 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:20,747 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-04T09:06:20.750 INFO:teuthology.orchestra.run.smithi006.stderr:==81160==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-04T09:06:20.756 INFO:teuthology.orchestra.run.smithi006.stderr:==81160==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe1e90f000; bottom 0x7f0dee9e0000; size: 0x00f02ff2f000 (1031596601344) 2023-12-04T09:06:20.757 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-04T09:06:20.757 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-04T09:06:21.293 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:21.293 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-04T09:06:21.293 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-04T09:06:21.293 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-04T09:06:21.293 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:21.293 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:21.348 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-04T09:06:21.349 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:21,348 seastar - Reactor backend: linux-aio 2023-12-04T09:06:21.367 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:21,366 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-04T09:06:21.367 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:21,366 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-04T09:06:21.367 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:21,366 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-04T09:06:21.374 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:21,374 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-04T09:06:21.377 INFO:teuthology.orchestra.run.smithi006.stderr:==81158==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-04T09:06:21.383 INFO:teuthology.orchestra.run.smithi006.stderr:==81158==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe1e90f000; bottom 0x7f0dec7d9000; size: 0x00f032136000 (1031632281600) 2023-12-04T09:06:21.383 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-04T09:06:21.383 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-04T09:06:21.409 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c922c6c 0x2c92401a 0x2c925568 0x2c9258ec 0x2c925a36 0x2c925e8c 0x54daf 0xb92c7 0x94469 0x94d46 0x22c42525 0x22c4284d 0x22cceeeb 0x22cd20fb 0x22d5d2e5 0x22d5fe2f 0x22d654a0 0x24c4328b 0x24c49f76 0x24c4c8c4 0x24c4da9d 0x227a99d9 0x227fa636 0x2281e36b 0x2c8c4f87 0x2c962d94 0x2cb872cd 0x2cb89117 0x2c5de54e 0x2c5e0457 0x1ed3586a 0x3feaf 0x3ff5f 0x1e7dea44 2023-12-04T09:06:21.409 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-04T09:06:21.468 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c922c6c 0x2c92401a 0x2c925568 0x2c9258ec 0x2c925a36 0x2c925e8c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x264a04ae 0x264a2af7 0x264a4908 0x264a716d 0x262d8055 0x1edf1f22 0x1ed63adf 0x1ed67920 0x2cca6e49 2023-12-04T09:06:21.468 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff862758f4 0xffffffff86275d8d 0xffffffff857508ea 0xffffffff8626646c 0xffffffff864000ea 2023-12-04T09:06:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-04T09:06:25.404 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-04T09:06:25.404 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-04T09:06:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:25.404 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:25.404 INFO:teuthology.orchestra.run.smithi006.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 1a67ff37-3baa-4e2c-8b80-4cd582792f60 2023-12-04T09:06:25.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-04T09:06:25.473 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-04T09:06:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,023 seastar - Reactor backend: linux-aio 2023-12-04T09:06:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,039 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-04T09:06:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,039 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-04T09:06:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,039 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-04T09:06:26.040 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,039 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-04T09:06:26.042 INFO:teuthology.orchestra.run.smithi006.stderr:==81342==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-04T09:06:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:==81342==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffddfeba000; bottom 0x7f4ac69e9000; size: 0x00b3194d1000 (769223626752) 2023-12-04T09:06:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-04T09:06:26.049 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-04T09:06:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-04T09:06:26.599 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-04T09:06:26.599 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-04T09:06:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:26.599 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-04T09:06:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,655 seastar - Reactor backend: linux-aio 2023-12-04T09:06:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,689 [shard 2] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-04T09:06:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,689 [shard 2] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-04T09:06:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,689 [shard 2] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-04T09:06:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:INFO 2023-12-04 09:06:26,689 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-04T09:06:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:==81340==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-04T09:06:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:==81340==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffddfeba000; bottom 0x7f4ac47e2000; size: 0x00b31b6d8000 (769259307008) 2023-12-04T09:06:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:False positive error reports may follow 2023-12-04T09:06:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-04T09:06:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c922c6c 0x2c92401a 0x2c925568 0x2c9258ec 0x2c925a36 0x2c925e8c 0x54daf 0x22cb3357 0x22cd28c4 0x22d5d2e5 0x22d5fe2f 0x22d654a0 0x24c4328b 0x24c49f76 0x24c4c8c4 0x24c4da9d 0x227a99d9 0x227fa636 0x2281e36b 0x2c8c4f87 0x2c962d94 0x2cb872cd 0x2cb89117 0x2c5de54e 0x2c5e0457 0x1ed3586a 0x3feaf 0x3ff5f 0x1e7dea44 2023-12-04T09:06:26.724 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 2023-12-04T09:06:26.784 INFO:teuthology.orchestra.run.smithi006.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c922c6c 0x2c92401a 0x2c925568 0x2c9258ec 0x2c925a36 0x2c925e8c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x264a04ae 0x264a2af7 0x264a4908 0x264a716d 0x262d8055 0x1edf1f22 0x1ed63adf 0x1ed67920 0x2cca6e49 2023-12-04T09:06:26.784 INFO:teuthology.orchestra.run.smithi006.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff862758f4 0xffffffff86275d8d 0xffffffff857508ea 0xffffffff8626646c 0xffffffff864000ea 2023-12-04T09:06:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:Suppressions used: 2023-12-04T09:06:30.720 INFO:teuthology.orchestra.run.smithi006.stderr: count bytes template 2023-12-04T09:06:30.720 INFO:teuthology.orchestra.run.smithi006.stderr: 1 8 MallocExtension::Register 2023-12-04T09:06:30.720 INFO:teuthology.orchestra.run.smithi006.stderr:----------------------------------------------------- 2023-12-04T09:06:30.720 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:30.721 INFO:teuthology.orchestra.run.smithi006.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 1a67ff37-3baa-4e2c-8b80-4cd582792f60 2023-12-04T09:06:30.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-04T09:06:30.790 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-04T09:06:30.790 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:30.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-04T09:06:30.861 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:30.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-04T09:06:30.931 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:30.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-04T09:06:31.001 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:31.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-04T09:06:31.072 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:31.072 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-04T09:06:31.127 INFO:tasks.ceph:Adding keys to all mons... 2023-12-04T09:06:31.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-04T09:06:31.196 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2023-12-04T09:06:31.196 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCElm1l7OL9GRAAiX6k3TTqiwQoOo3Oq2oFZw== 2023-12-04T09:06:31.196 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2023-12-04T09:06:31.196 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCIlm1lV8WgBxAAKQpKNbCUBCgzv8WQYGuS9Q== 2023-12-04T09:06:31.196 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2023-12-04T09:06:31.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCNlm1lanrPGRAAYV1wNyINbe0hIz12twG8Sg== 2023-12-04T09:06:31.197 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2023-12-04T09:06:31.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCSlm1lV3uiLBAAC97iptNmtKM/Kc2mRo7i5Q== 2023-12-04T09:06:31.197 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2023-12-04T09:06:31.197 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCElm1lpPA8HRAAlXXiO3CFrpBIZfDh1ytH4A== 2023-12-04T09:06:31.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T09:06:31.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T09:06:31.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T09:06:31.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T09:06:31.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T09:06:31.548 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-04T09:06:31.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-04T09:06:31.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T09:06:31.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:31.682+0000 7fede5c22980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:31.683+0000 7fede5c22980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.700 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-04T09:06:31.730 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-04T09:06:31.784 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-04T09:06:31.785 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-04T09:06:31.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-04T09:06:31.827 INFO:tasks.ceph.mon.a:Started 2023-12-04T09:06:31.827 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-04T09:06:31.827 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-04T09:06:31.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-04T09:06:31.829 INFO:tasks.ceph.mgr.x:Started 2023-12-04T09:06:31.829 INFO:tasks.ceph:Setting crush tunables to default 2023-12-04T09:06:31.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2023-12-04T09:06:31.914 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-04T09:06:31.912+0000 7fe2e802a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.914 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-04T09:06:31.912+0000 7fe2e802a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.914 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-04T09:06:31.912+0000 7fe2e802a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.914 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:31.913+0000 7f8c43820f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:31.938+0000 7fb82c460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:31.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:31.965+0000 7fb82c460640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:32.037 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2023-12-04T09:06:32.048 INFO:tasks.ceph:check_enable_crimson: True 2023-12-04T09:06:32.048 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-12-04T09:06:32.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-12-04T09:06:32.118 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:32.116+0000 7f8c43820f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:32.118 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:32.117+0000 7f8c43820f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:32.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:32.153+0000 7f1e3aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:32.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:32.159+0000 7f1e3aa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:32.242 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-12-04T09:06:32.240+0000 7fe2dfff1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-12-04T09:06:32.242 INFO:tasks.ceph.mon.a.smithi006.stderr:Please be aware that this feature is experimental, untested, 2023-12-04T09:06:32.242 INFO:tasks.ceph.mon.a.smithi006.stderr:unsupported, and may result in data corruption, data loss, 2023-12-04T09:06:32.242 INFO:tasks.ceph.mon.a.smithi006.stderr:and/or irreparable damage to your cluster. Do not use 2023-12-04T09:06:32.242 INFO:tasks.ceph.mon.a.smithi006.stderr:feature with important data. 2023-12-04T09:06:32.242 INFO:tasks.ceph.mon.a.smithi006.stderr: 2023-12-04T09:06:32.581 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:32.580+0000 7f8c43820f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T09:06:32.635 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:32.634+0000 7f8c43820f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T09:06:33.050 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-04T09:06:33.050 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:33.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-04T09:06:33.081 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:33.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-04T09:06:33.153 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:33.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-04T09:06:33.205 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.203+0000 7f8c43820f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T09:06:33.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd new 2eba799f-5403-4b00-8caf-4cbfbe1a32bf 0 2023-12-04T09:06:33.261 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.260+0000 7f8c43820f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T09:06:33.310 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.309+0000 7f8c43820f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T09:06:33.369 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.367+0000 7f8c43820f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T09:06:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:33.366+0000 7fbcc806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:33.373+0000 7fbcc806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:33.414 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.412+0000 7f8c43820f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T09:06:33.453 INFO:teuthology.orchestra.run.smithi006.stdout:0 2023-12-04T09:06:33.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd new 7bc3554c-46cb-4267-b93d-cfc03cd57f3e 1 2023-12-04T09:06:33.523 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.521+0000 7f8c43820f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-04T09:06:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:33.569+0000 7f21cb406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:33.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:33.576+0000 7f21cb406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:33.599 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.597+0000 7f8c43820f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T09:06:33.647 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.645+0000 7f8c43820f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T09:06:33.656 INFO:teuthology.orchestra.run.smithi006.stdout:1 2023-12-04T09:06:33.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd new b4d95557-07bc-46d3-92aa-8b8459f3f3e3 2 2023-12-04T09:06:33.704 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.702+0000 7f8c43820f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T09:06:33.755 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:33.754+0000 7f8c43820f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T09:06:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:33.772+0000 7fb84162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:33.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:33.779+0000 7fb84162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:33.859 INFO:teuthology.orchestra.run.smithi006.stdout:2 2023-12-04T09:06:33.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-12-04T09:06:33.946 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-04T09:06:33.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-12-04T09:06:33.989 INFO:tasks.ceph.osd.0:Started 2023-12-04T09:06:33.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-12-04T09:06:34.024 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-04T09:06:34.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-12-04T09:06:34.067 INFO:tasks.ceph.osd.1:Started 2023-12-04T09:06:34.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-12-04T09:06:34.103 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-04T09:06:34.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-12-04T09:06:34.146 INFO:tasks.ceph.osd.2:Started 2023-12-04T09:06:34.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T09:06:34.181 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:34.179+0000 7f8c43820f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-04T09:06:34.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:34.347+0000 7efc0235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:34.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:34.358+0000 7efc0235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:34.433 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:34.431+0000 7f8c43820f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T09:06:34.499 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:34.499 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":6,"fsid":"1a67ff37-3baa-4e2c-8b80-4cd582792f60","created":"2023-12-04T09:06:31.958334+0000","modified":"2023-12-04T09:06:33.857766+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T09:06:33.857766+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2eba799f-5403-4b00-8caf-4cbfbe1a32bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7bc3554c-46cb-4267-b93d-cfc03cd57f3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b4d95557-07bc-46d3-92aa-8b8459f3f3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T09:06:34.520 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-04T09:06:34.520 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-04T09:06:34.607 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:34.605+0000 7f8c43820f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T09:06:34.658 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:34.656+0000 7f8c43820f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T09:06:34.763 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:34.762+0000 7f8c43820f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T09:06:34.822 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T09:06:34.869 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:34.867+0000 7f8c43820f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T09:06:34.926 INFO:teuthology.misc.health.smithi006.stderr:2023-12-04T09:06:34.920+0000 7fb39099e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:34.928 INFO:teuthology.misc.health.smithi006.stderr:2023-12-04T09:06:34.926+0000 7fb39099e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:34.995 INFO:teuthology.misc.health.smithi006.stdout: 2023-12-04T09:06:34.995 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":6,"fsid":"1a67ff37-3baa-4e2c-8b80-4cd582792f60","created":"2023-12-04T09:06:31.958334+0000","modified":"2023-12-04T09:06:33.857766+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T09:06:33.857766+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2eba799f-5403-4b00-8caf-4cbfbe1a32bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7bc3554c-46cb-4267-b93d-cfc03cd57f3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b4d95557-07bc-46d3-92aa-8b8459f3f3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T09:06:35.006 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-04T09:06:35.028 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:35.027+0000 7f8c43820f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T09:06:35.075 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:35.074+0000 7f8c43820f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T09:06:35.231 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:35.229+0000 7f8c43820f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T09:06:36.088 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:36.086+0000 7f8c43820f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T09:06:36.401 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:36.400+0000 7f8c43820f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T09:06:36.457 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:36.456+0000 7f8c43820f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T09:06:36.571 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:36.569+0000 7f8c43820f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T09:06:37.590 INFO:tasks.ceph.mgr.x.smithi006.stderr:2023-12-04T09:06:37.589+0000 7f8be3915640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-04T09:06:41.308 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T09:06:41.419 INFO:teuthology.misc.health.smithi006.stderr:2023-12-04T09:06:41.414+0000 7f1e104ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:41.421 INFO:teuthology.misc.health.smithi006.stderr:2023-12-04T09:06:41.419+0000 7f1e104ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:41.607 INFO:teuthology.misc.health.smithi006.stdout: 2023-12-04T09:06:41.608 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":9,"fsid":"1a67ff37-3baa-4e2c-8b80-4cd582792f60","created":"2023-12-04T09:06:31.958334+0000","modified":"2023-12-04T09:06:41.596829+0000","last_up_change":"2023-12-04T09:06:39.590800+0000","last_in_change":"2023-12-04T09:06:33.857766+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T09:06:40.643965+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2eba799f-5403-4b00-8caf-4cbfbe1a32bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":26497944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":26497944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":26497944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":26497944}]},"public_addr":"172.21.15.6:6803/26497944","cluster_addr":"172.21.15.6:6802/26497944","heartbeat_back_addr":"172.21.15.6:6809/26497944","heartbeat_front_addr":"172.21.15.6:6810/26497944","state":["exists","up"]},{"osd":1,"uuid":"7bc3554c-46cb-4267-b93d-cfc03cd57f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":859278825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":859278825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":859278825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":859278825}]},"public_addr":"172.21.15.6:6805/859278825","cluster_addr":"172.21.15.6:6804/859278825","heartbeat_back_addr":"172.21.15.6:6811/859278825","heartbeat_front_addr":"172.21.15.6:6808/859278825","state":["exists","up"]},{"osd":2,"uuid":"b4d95557-07bc-46d3-92aa-8b8459f3f3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":626241314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":626241314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":626241314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":626241314}]},"public_addr":"172.21.15.6:6806/626241314","cluster_addr":"172.21.15.6:6807/626241314","heartbeat_back_addr":"172.21.15.6:6812/626241314","heartbeat_front_addr":"172.21.15.6:6813/626241314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T09:06:41.620 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-04T09:06:41.620 INFO:tasks.ceph:Creating RBD pool 2023-12-04T09:06:41.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-04T09:06:41.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:41.724+0000 7f29fe95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:41.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:41.732+0000 7f29fe95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:42.604 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2023-12-04T09:06:42.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-04T09:06:42.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:42.741+0000 7fa4af75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:42.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:42.752+0000 7fa4af75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:43.611 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-04T09:06:43.625 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-04T09:06:43.625 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-04T09:06:43.625 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-04T09:06:43.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-04T09:06:43.626 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-04T09:06:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:43.744+0000 7fd70706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:43.749+0000 7fd70706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.003 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:44.015 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2649285397},{"type":"v1","addr":"172.21.15.6:6801","nonce":2649285397}]},"active_addr":"172.21.15.6:6801/2649285397","active_change":"2023-12-04T09:06:36.582620+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1274797876}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":97263661}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1627998984}]}]} 2023-12-04T09:06:44.017 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-04T09:06:44.018 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-04T09:06:44.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T09:06:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.124+0000 7f927f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.129+0000 7f927f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.315 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:44.315 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":11,"fsid":"1a67ff37-3baa-4e2c-8b80-4cd582792f60","created":"2023-12-04T09:06:31.958334+0000","modified":"2023-12-04T09:06:43.607260+0000","last_up_change":"2023-12-04T09:06:39.590800+0000","last_in_change":"2023-12-04T09:06:33.857766+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T09:06:40.643965+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-04T09:06:41.944019+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2eba799f-5403-4b00-8caf-4cbfbe1a32bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":26497944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":26497944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":26497944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":26497944}]},"public_addr":"172.21.15.6:6803/26497944","cluster_addr":"172.21.15.6:6802/26497944","heartbeat_back_addr":"172.21.15.6:6809/26497944","heartbeat_front_addr":"172.21.15.6:6810/26497944","state":["exists","up"]},{"osd":1,"uuid":"7bc3554c-46cb-4267-b93d-cfc03cd57f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":859278825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":859278825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":859278825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":859278825}]},"public_addr":"172.21.15.6:6805/859278825","cluster_addr":"172.21.15.6:6804/859278825","heartbeat_back_addr":"172.21.15.6:6811/859278825","heartbeat_front_addr":"172.21.15.6:6808/859278825","state":["exists","up"]},{"osd":2,"uuid":"b4d95557-07bc-46d3-92aa-8b8459f3f3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":626241314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":626241314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":626241314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":626241314}]},"public_addr":"172.21.15.6:6806/626241314","cluster_addr":"172.21.15.6:6807/626241314","heartbeat_back_addr":"172.21.15.6:6812/626241314","heartbeat_front_addr":"172.21.15.6:6813/626241314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T09:06:44.327 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-04T09:06:44.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T09:06:44.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.433+0000 7fa352b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.438+0000 7fa352b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.626 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:44.626 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":12,"fsid":"1a67ff37-3baa-4e2c-8b80-4cd582792f60","created":"2023-12-04T09:06:31.958334+0000","modified":"2023-12-04T09:06:44.610460+0000","last_up_change":"2023-12-04T09:06:39.590800+0000","last_in_change":"2023-12-04T09:06:33.857766+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T09:06:40.643965+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-04T09:06:41.944019+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2eba799f-5403-4b00-8caf-4cbfbe1a32bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":26497944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":26497944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":26497944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":26497944}]},"public_addr":"172.21.15.6:6803/26497944","cluster_addr":"172.21.15.6:6802/26497944","heartbeat_back_addr":"172.21.15.6:6809/26497944","heartbeat_front_addr":"172.21.15.6:6810/26497944","state":["exists","up"]},{"osd":1,"uuid":"7bc3554c-46cb-4267-b93d-cfc03cd57f3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":859278825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":859278825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":859278825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":859278825}]},"public_addr":"172.21.15.6:6805/859278825","cluster_addr":"172.21.15.6:6804/859278825","heartbeat_back_addr":"172.21.15.6:6811/859278825","heartbeat_front_addr":"172.21.15.6:6808/859278825","state":["exists","up"]},{"osd":2,"uuid":"b4d95557-07bc-46d3-92aa-8b8459f3f3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":626241314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":626241314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":626241314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":626241314}]},"public_addr":"172.21.15.6:6806/626241314","cluster_addr":"172.21.15.6:6807/626241314","heartbeat_back_addr":"172.21.15.6:6812/626241314","heartbeat_front_addr":"172.21.15.6:6813/626241314","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T09:06:44.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-04T09:06:44.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-04T09:06:44.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-04T09:06:44.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.755+0000 7febb719e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.755+0000 7f14a339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.756+0000 7f3a1919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.762+0000 7f3a1919e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.762+0000 7febb719e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.762+0000 7f14a339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.790 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-04T09:06:44.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-04T09:06:44.791 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-04T09:06:44.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-04T09:06:44.792 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-04T09:06:44.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-04T09:06:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.902+0000 7ff7d8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.910+0000 7ff7d8e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.944+0000 7f847fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.951+0000 7f847fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.946+0000 7f43a1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:44.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:44.959+0000 7f43a1e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:45.111 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-04T09:06:45.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-12-04T09:06:45.138 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-04T09:06:45.150 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-12-04T09:06:45.151 DEBUG:teuthology.parallel:result is None 2023-12-04T09:06:45.154 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-04T09:06:45.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-12-04T09:06:46.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-04T09:06:46.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-04T09:06:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:46.249+0000 7f161486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:46.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:46.257+0000 7f161486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:46.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:46.289+0000 7fb373d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:46.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:46.295+0000 7fb373d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:46.445 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-04T09:06:46.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-12-04T09:06:46.484 INFO:teuthology.orchestra.run.smithi006.stdout:34359738372 2023-12-04T09:06:46.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-12-04T09:06:47.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-04T09:06:47.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-04T09:06:47.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:47.581+0000 7f1217c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:47.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:47.589+0000 7f1217c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:47.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:47.618+0000 7fe2b799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:47.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:47.625+0000 7fe2b799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:47.780 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-04T09:06:47.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-12-04T09:06:47.793 DEBUG:teuthology.parallel:result is None 2023-12-04T09:06:47.817 INFO:teuthology.orchestra.run.smithi006.stdout:34359738373 2023-12-04T09:06:47.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-12-04T09:06:47.830 DEBUG:teuthology.parallel:result is None 2023-12-04T09:06:47.830 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-04T09:06:47.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T09:06:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:47.978+0000 7fcdaf82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:47.983+0000 7fcdaf82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:48.167 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-12-04T09:06:48.179 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-04T09:06:46.587723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.948742"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.653745+0000","last_change":"2023-12-04T09:06:43.732631+0000","last_active":"2023-12-04T09:06:44.653745+0000","last_peered":"2023-12-04T09:06:44.653745+0000","last_clean":"2023-12-04T09:06:44.653745+0000","last_became_active":"2023-12-04T09:06:43.730803+0000","last_became_peered":"2023-12-04T09:06:43.730803+0000","last_unstale":"2023-12-04T09:06:44.653745+0000","last_undegraded":"2023-12-04T09:06:44.653745+0000","last_fullsized":"2023-12-04T09:06:44.653745+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.668198+0000","last_change":"2023-12-04T09:06:43.734425+0000","last_active":"2023-12-04T09:06:44.668198+0000","last_peered":"2023-12-04T09:06:44.668198+0000","last_clean":"2023-12-04T09:06:44.668198+0000","last_became_active":"2023-12-04T09:06:43.732646+0000","last_became_peered":"2023-12-04T09:06:43.732646+0000","last_unstale":"2023-12-04T09:06:44.668198+0000","last_undegraded":"2023-12-04T09:06:44.668198+0000","last_fullsized":"2023-12-04T09:06:44.668198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.660832+0000","last_change":"2023-12-04T09:06:43.755468+0000","last_active":"2023-12-04T09:06:44.660832+0000","last_peered":"2023-12-04T09:06:44.660832+0000","last_clean":"2023-12-04T09:06:44.660832+0000","last_became_active":"2023-12-04T09:06:43.746653+0000","last_became_peered":"2023-12-04T09:06:43.746653+0000","last_unstale":"2023-12-04T09:06:44.660832+0000","last_undegraded":"2023-12-04T09:06:44.660832+0000","last_fullsized":"2023-12-04T09:06:44.660832+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.662316+0000","last_change":"2023-12-04T09:06:43.765455+0000","last_active":"2023-12-04T09:06:44.662316+0000","last_peered":"2023-12-04T09:06:44.662316+0000","last_clean":"2023-12-04T09:06:44.662316+0000","last_became_active":"2023-12-04T09:06:43.760377+0000","last_became_peered":"2023-12-04T09:06:43.760377+0000","last_unstale":"2023-12-04T09:06:44.662316+0000","last_undegraded":"2023-12-04T09:06:44.662316+0000","last_fullsized":"2023-12-04T09:06:44.662316+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.673094+0000","last_change":"2023-12-04T09:06:43.714989+0000","last_active":"2023-12-04T09:06:44.673094+0000","last_peered":"2023-12-04T09:06:44.673094+0000","last_clean":"2023-12-04T09:06:44.673094+0000","last_became_active":"2023-12-04T09:06:43.713170+0000","last_became_peered":"2023-12-04T09:06:43.713170+0000","last_unstale":"2023-12-04T09:06:44.673094+0000","last_undegraded":"2023-12-04T09:06:44.673094+0000","last_fullsized":"2023-12-04T09:06:44.673094+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.620325+0000","last_change":"2023-12-04T09:06:43.720828+0000","last_active":"2023-12-04T09:06:44.620325+0000","last_peered":"2023-12-04T09:06:44.620325+0000","last_clean":"2023-12-04T09:06:44.620325+0000","last_became_active":"2023-12-04T09:06:43.719036+0000","last_became_peered":"2023-12-04T09:06:43.719036+0000","last_unstale":"2023-12-04T09:06:44.620325+0000","last_undegraded":"2023-12-04T09:06:44.620325+0000","last_fullsized":"2023-12-04T09:06:44.620325+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.619105+0000","last_change":"2023-12-04T09:06:43.707655+0000","last_active":"2023-12-04T09:06:44.619105+0000","last_peered":"2023-12-04T09:06:44.619105+0000","last_clean":"2023-12-04T09:06:44.619105+0000","last_became_active":"2023-12-04T09:06:43.704766+0000","last_became_peered":"2023-12-04T09:06:43.704766+0000","last_unstale":"2023-12-04T09:06:44.619105+0000","last_undegraded":"2023-12-04T09:06:44.619105+0000","last_fullsized":"2023-12-04T09:06:44.619105+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.669203+0000","last_change":"2023-12-04T09:06:43.692893+0000","last_active":"2023-12-04T09:06:44.669203+0000","last_peered":"2023-12-04T09:06:44.669203+0000","last_clean":"2023-12-04T09:06:44.669203+0000","last_became_active":"2023-12-04T09:06:43.690923+0000","last_became_peered":"2023-12-04T09:06:43.690923+0000","last_unstale":"2023-12-04T09:06:44.669203+0000","last_undegraded":"2023-12-04T09:06:44.669203+0000","last_fullsized":"2023-12-04T09:06:44.669203+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.653553+0000","last_change":"2023-12-04T09:06:42.685334+0000","last_active":"2023-12-04T09:06:44.653553+0000","last_peered":"2023-12-04T09:06:44.653553+0000","last_clean":"2023-12-04T09:06:44.653553+0000","last_became_active":"2023-12-04T09:06:42.677805+0000","last_became_peered":"2023-12-04T09:06:42.677805+0000","last_unstale":"2023-12-04T09:06:44.653553+0000","last_undegraded":"2023-12-04T09:06:44.653553+0000","last_fullsized":"2023-12-04T09:06:44.653553+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:41.596829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:41.596829+0000","last_clean_scrub_stamp":"2023-12-04T09:06:41.596829+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-04T09:06:48.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T09:06:48.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:48.286+0000 7f1219d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:48.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:48.291+0000 7f1219d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:48.472 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-12-04T09:06:48.484 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-04T09:06:46.587723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59924,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.948742"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.653745+0000","last_change":"2023-12-04T09:06:43.732631+0000","last_active":"2023-12-04T09:06:44.653745+0000","last_peered":"2023-12-04T09:06:44.653745+0000","last_clean":"2023-12-04T09:06:44.653745+0000","last_became_active":"2023-12-04T09:06:43.730803+0000","last_became_peered":"2023-12-04T09:06:43.730803+0000","last_unstale":"2023-12-04T09:06:44.653745+0000","last_undegraded":"2023-12-04T09:06:44.653745+0000","last_fullsized":"2023-12-04T09:06:44.653745+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.668198+0000","last_change":"2023-12-04T09:06:43.734425+0000","last_active":"2023-12-04T09:06:44.668198+0000","last_peered":"2023-12-04T09:06:44.668198+0000","last_clean":"2023-12-04T09:06:44.668198+0000","last_became_active":"2023-12-04T09:06:43.732646+0000","last_became_peered":"2023-12-04T09:06:43.732646+0000","last_unstale":"2023-12-04T09:06:44.668198+0000","last_undegraded":"2023-12-04T09:06:44.668198+0000","last_fullsized":"2023-12-04T09:06:44.668198+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.660832+0000","last_change":"2023-12-04T09:06:43.755468+0000","last_active":"2023-12-04T09:06:44.660832+0000","last_peered":"2023-12-04T09:06:44.660832+0000","last_clean":"2023-12-04T09:06:44.660832+0000","last_became_active":"2023-12-04T09:06:43.746653+0000","last_became_peered":"2023-12-04T09:06:43.746653+0000","last_unstale":"2023-12-04T09:06:44.660832+0000","last_undegraded":"2023-12-04T09:06:44.660832+0000","last_fullsized":"2023-12-04T09:06:44.660832+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.662316+0000","last_change":"2023-12-04T09:06:43.765455+0000","last_active":"2023-12-04T09:06:44.662316+0000","last_peered":"2023-12-04T09:06:44.662316+0000","last_clean":"2023-12-04T09:06:44.662316+0000","last_became_active":"2023-12-04T09:06:43.760377+0000","last_became_peered":"2023-12-04T09:06:43.760377+0000","last_unstale":"2023-12-04T09:06:44.662316+0000","last_undegraded":"2023-12-04T09:06:44.662316+0000","last_fullsized":"2023-12-04T09:06:44.662316+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.673094+0000","last_change":"2023-12-04T09:06:43.714989+0000","last_active":"2023-12-04T09:06:44.673094+0000","last_peered":"2023-12-04T09:06:44.673094+0000","last_clean":"2023-12-04T09:06:44.673094+0000","last_became_active":"2023-12-04T09:06:43.713170+0000","last_became_peered":"2023-12-04T09:06:43.713170+0000","last_unstale":"2023-12-04T09:06:44.673094+0000","last_undegraded":"2023-12-04T09:06:44.673094+0000","last_fullsized":"2023-12-04T09:06:44.673094+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.620325+0000","last_change":"2023-12-04T09:06:43.720828+0000","last_active":"2023-12-04T09:06:44.620325+0000","last_peered":"2023-12-04T09:06:44.620325+0000","last_clean":"2023-12-04T09:06:44.620325+0000","last_became_active":"2023-12-04T09:06:43.719036+0000","last_became_peered":"2023-12-04T09:06:43.719036+0000","last_unstale":"2023-12-04T09:06:44.620325+0000","last_undegraded":"2023-12-04T09:06:44.620325+0000","last_fullsized":"2023-12-04T09:06:44.620325+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.619105+0000","last_change":"2023-12-04T09:06:43.707655+0000","last_active":"2023-12-04T09:06:44.619105+0000","last_peered":"2023-12-04T09:06:44.619105+0000","last_clean":"2023-12-04T09:06:44.619105+0000","last_became_active":"2023-12-04T09:06:43.704766+0000","last_became_peered":"2023-12-04T09:06:43.704766+0000","last_unstale":"2023-12-04T09:06:44.619105+0000","last_undegraded":"2023-12-04T09:06:44.619105+0000","last_fullsized":"2023-12-04T09:06:44.619105+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.669203+0000","last_change":"2023-12-04T09:06:43.692893+0000","last_active":"2023-12-04T09:06:44.669203+0000","last_peered":"2023-12-04T09:06:44.669203+0000","last_clean":"2023-12-04T09:06:44.669203+0000","last_became_active":"2023-12-04T09:06:43.690923+0000","last_became_peered":"2023-12-04T09:06:43.690923+0000","last_unstale":"2023-12-04T09:06:44.669203+0000","last_undegraded":"2023-12-04T09:06:44.669203+0000","last_fullsized":"2023-12-04T09:06:44.669203+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:42.602509+0000","last_clean_scrub_stamp":"2023-12-04T09:06:42.602509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-04T09:06:44.653553+0000","last_change":"2023-12-04T09:06:42.685334+0000","last_active":"2023-12-04T09:06:44.653553+0000","last_peered":"2023-12-04T09:06:44.653553+0000","last_clean":"2023-12-04T09:06:44.653553+0000","last_became_active":"2023-12-04T09:06:42.677805+0000","last_became_peered":"2023-12-04T09:06:42.677805+0000","last_unstale":"2023-12-04T09:06:44.653553+0000","last_undegraded":"2023-12-04T09:06:44.653553+0000","last_fullsized":"2023-12-04T09:06:44.653553+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T09:06:41.596829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T09:06:41.596829+0000","last_clean_scrub_stamp":"2023-12-04T09:06:41.596829+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29962,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-04T09:06:48.485 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-04T09:06:48.485 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-04T09:06:48.485 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-04T09:06:48.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T09:06:48.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:48.592+0000 7f2cde6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:06:48.596+0000 7f2cde6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:06:48.818 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:48.818 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-04T09:06:48.830 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-04T09:06:48.830 INFO:teuthology.run_tasks:Running task cbt... 2023-12-04T09:06:48.842 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi006.front.sepia.ceph.com', 'osds': ['ubuntu@smithi006.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi006.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi006.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi006.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-12-04T09:06:48.842 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-12-04T09:06:48.862 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:06:48.862 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-04T09:06:48.912 DEBUG:teuthology.orchestra.run.smithi006:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-12-04T09:06:48.967 INFO:teuthology.orchestra.run.smithi006.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-12-04T09:06:49.487 INFO:tasks.cbt:Installing collectl 2023-12-04T09:06:49.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-12-04T09:06:49.593 INFO:teuthology.orchestra.run.smithi006.stderr:--2023-12-04 09:06:49-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-12-04T09:06:49.737 INFO:teuthology.orchestra.run.smithi006.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2023-12-04T09:06:49.750 INFO:teuthology.orchestra.run.smithi006.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-12-04T09:06:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-04T09:06:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlbZapRrOm5zLmL4l6B1hTkDE0DGu-8SuHUEKlFbCKRuSiA0fq540T-miIueKG6hQF5yABIO_xLDNArqT6g97aPylprw%3D%3D&use_mirror=gigenet&r= [following] 2023-12-04T09:06:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:--2023-12-04 09:06:50-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlbZapRrOm5zLmL4l6B1hTkDE0DGu-8SuHUEKlFbCKRuSiA0fq540T-miIueKG6hQF5yABIO_xLDNArqT6g97aPylprw%3D%3D&use_mirror=gigenet&r= 2023-12-04T09:06:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-12-04T09:06:50.149 INFO:teuthology.orchestra.run.smithi006.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-12-04T09:06:51.167 INFO:teuthology.orchestra.run.smithi006.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-04T09:06:51.168 INFO:teuthology.orchestra.run.smithi006.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-12-04T09:06:51.168 INFO:teuthology.orchestra.run.smithi006.stderr:--2023-12-04 09:06:51-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-12-04T09:06:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2023-12-04T09:06:51.209 INFO:teuthology.orchestra.run.smithi006.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2023-12-04T09:06:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:HTTP request sent, awaiting response... 200 OK 2023-12-04T09:06:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:Length: 620693 (606K) [application/x-gzip] 2023-12-04T09:06:51.460 INFO:teuthology.orchestra.run.smithi006.stderr:Saving to: ‘collectl.tar.gz’ 2023-12-04T09:06:51.460 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:51.515 INFO:teuthology.orchestra.run.smithi006.stderr: 0K .......... .......... .......... .......... .......... 8% 902K 1s 2023-12-04T09:06:51.543 INFO:teuthology.orchestra.run.smithi006.stderr: 50K .......... .......... .......... .......... .......... 16% 1.72M 0s 2023-12-04T09:06:51.571 INFO:teuthology.orchestra.run.smithi006.stderr: 100K .......... .......... .......... .......... .......... 24% 1.75M 0s 2023-12-04T09:06:51.600 INFO:teuthology.orchestra.run.smithi006.stderr: 150K .......... .......... .......... .......... .......... 32% 1.74M 0s 2023-12-04T09:06:51.600 INFO:teuthology.orchestra.run.smithi006.stderr: 200K .......... .......... .......... .......... .......... 41% 153M 0s 2023-12-04T09:06:51.628 INFO:teuthology.orchestra.run.smithi006.stderr: 250K .......... .......... .......... .......... .......... 49% 1.78M 0s 2023-12-04T09:06:51.629 INFO:teuthology.orchestra.run.smithi006.stderr: 300K .......... .......... .......... .......... .......... 57% 89.6M 0s 2023-12-04T09:06:51.655 INFO:teuthology.orchestra.run.smithi006.stderr: 350K .......... .......... .......... .......... .......... 65% 1.78M 0s 2023-12-04T09:06:51.656 INFO:teuthology.orchestra.run.smithi006.stderr: 400K .......... .......... .......... .......... .......... 74% 50.7M 0s 2023-12-04T09:06:51.657 INFO:teuthology.orchestra.run.smithi006.stderr: 450K .......... .......... .......... .......... .......... 82% 62.2M 0s 2023-12-04T09:06:51.684 INFO:teuthology.orchestra.run.smithi006.stderr: 500K .......... .......... .......... .......... .......... 90% 1.82M 0s 2023-12-04T09:06:51.684 INFO:teuthology.orchestra.run.smithi006.stderr: 550K .......... .......... .......... .......... .......... 98% 78.5M 0s 2023-12-04T09:06:51.684 INFO:teuthology.orchestra.run.smithi006.stderr: 600K ...... 100% 11.4T=0.2s 2023-12-04T09:06:51.685 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:51.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04 09:06:51 (2.64 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-12-04T09:06:51.685 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:06:51.740 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/ 2023-12-04T09:06:51.740 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/collectl.conf 2023-12-04T09:06:51.740 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/ 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/SlowProc.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/WhySummary.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Socket.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/colmux.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Tutorial.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Gexpr.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Import.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/style.css 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/FileNaming.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/TheMath.html 2023-12-04T09:06:51.741 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Environmental.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Process.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/HiResTime.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/slub.jpg 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Lustre.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Documentation.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Inodes.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Graphite.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/InputFiles.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Export.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Interrupts.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Startup.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Performance.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Messages.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Logging.html 2023-12-04T09:06:51.742 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data-detail.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/CPUs.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Matrix.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data-brief.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Misc.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Examples.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OpenStack.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Data.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Exceptions.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-12-04T09:06:51.743 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Playback.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Hello.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Sockets.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Network.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Infiniband.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Memory.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/Disks.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/service/ 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/service/collectl.service 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/ 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl-suse 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl-debian 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl-generic 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/initd/collectl 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/misc.ph 2023-12-04T09:06:51.744 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/gexpr.ph 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/statsd.ph 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/envrules.std 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/hello.ph 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/vmsum.ph 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/proctree.ph 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/README-WINDOWS 2023-12-04T09:06:51.745 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/formatit.ph 2023-12-04T09:06:51.746 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/colmux 2023-12-04T09:06:51.746 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/INSTALL 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/graphite.ph 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/RELEASE-collectl 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/vmstat.ph 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/README 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/man1/ 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/man1/collectl.1 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/man1/colmux.1 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/lexpr.ph 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/client.pl 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/collectl 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/COPYING 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/vnet.ph 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/GPL 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/UNINSTALL 2023-12-04T09:06:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:collectl-4.3.1/ARTISTIC 2023-12-04T09:06:51.807 INFO:teuthology.orchestra.run.smithi006.stderr:./INSTALL: line 90: chkconfig: command not found 2023-12-04T09:06:51.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-12-04T09:06:52.423 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:06:43 ago on Mon 04 Dec 2023 09:00:09 AM UTC. 2023-12-04T09:06:52.523 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-12-04T09:06:52.525 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-12-04T09:06:52.583 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-12-04T09:06:52.583 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: librbd-devel x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 23 k 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: perf x86_64 5.14.0-388.el9 appstream 7.6 M 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: libradospp-devel x86_64 2:18.0.0-7693.gebb1a6e3.el9 ceph 30 k 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout:Install 5 Packages 2023-12-04T09:06:52.584 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:52.585 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 7.7 M 2023-12-04T09:06:52.585 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 11 M 2023-12-04T09:06:52.585 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-12-04T09:06:52.873 INFO:teuthology.orchestra.run.smithi006.stdout:(1/5): librbd-devel-18.0.0-7693.gebb1a6e3.el9.x 141 kB/s | 23 kB 00:00 2023-12-04T09:06:52.890 INFO:teuthology.orchestra.run.smithi006.stdout:(2/5): libradospp-devel-18.0.0-7693.gebb1a6e3.e 167 kB/s | 30 kB 00:00 2023-12-04T09:06:53.065 INFO:teuthology.orchestra.run.smithi006.stdout:(3/5): perf-5.14.0-388.el9.x86_64.rpm 21 MB/s | 7.6 MB 00:00 2023-12-04T09:06:53.082 INFO:teuthology.orchestra.run.smithi006.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 68 kB/s | 13 kB 00:00 2023-12-04T09:06:53.282 INFO:teuthology.orchestra.run.smithi006.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 290 kB/s | 118 kB 00:00 2023-12-04T09:06:53.283 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-12-04T09:06:53.283 INFO:teuthology.orchestra.run.smithi006.stdout:Total 11 MB/s | 7.7 MB 00:00 2023-12-04T09:06:53.365 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-12-04T09:06:53.389 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-12-04T09:06:53.389 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-12-04T09:06:53.608 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-12-04T09:06:53.608 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-12-04T09:06:54.003 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-12-04T09:06:54.077 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-12-04T09:06:54.106 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-12-04T09:06:54.138 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradospp-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 3/5 2023-12-04T09:06:54.496 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librbd-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 4/5 2023-12-04T09:06:54.621 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : perf-5.14.0-388.el9.x86_64 5/5 2023-12-04T09:06:55.616 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: perf-5.14.0-388.el9.x86_64 5/5 2023-12-04T09:06:55.616 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradospp-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 1/5 2023-12-04T09:06:55.616 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2/5 2023-12-04T09:06:55.616 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : perf-5.14.0-388.el9.x86_64 3/5 2023-12-04T09:06:55.616 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: libradospp-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: librbd-devel-2:18.0.0-7693.gebb1a6e3.el9.x86_64 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh-2.34-7.el9.x86_64 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: perf-5.14.0-388.el9.x86_64 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-12-04T09:06:56.002 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-12-04T09:06:56.344 INFO:tasks.cbt:benchmark: radosbench 2023-12-04T09:06:56.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-12-04T09:06:56.376 DEBUG:teuthology.orchestra.run.smithi006:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - Settings.cluster: 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'clients': ['ubuntu@smithi006.front.sepia.ceph.com'], 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'head': 'ubuntu@smithi006.front.sepia.ceph.com', 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'is_teuthology': True, 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'iterations': 1, 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'mons': ['ubuntu@smithi006.front.sepia.ceph.com'], 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'osds': ['ubuntu@smithi006.front.sepia.ceph.com'], 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'osds_per_node': 3, 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'pid_dir': '/var/run/ceph', 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'pgp_size': 256, 2023-12-04T09:06:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 'replication': 'replicated'}}, 2023-12-04T09:06:56.808 INFO:teuthology.orchestra.run.smithi006.stderr: 'rebuild_every_test': False, 2023-12-04T09:06:56.808 INFO:teuthology.orchestra.run.smithi006.stderr: 'rgws': {'ubuntu@smithi006.front.sepia.ceph.com': None}, 2023-12-04T09:06:56.808 INFO:teuthology.orchestra.run.smithi006.stderr: 'tmp_dir': '/tmp/cbt', 2023-12-04T09:06:56.808 INFO:teuthology.orchestra.run.smithi006.stderr: 'use_existing': True, 2023-12-04T09:06:56.808 INFO:teuthology.orchestra.run.smithi006.stderr: 'user': 'ubuntu'} 2023-12-04T09:06:56.829 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-12-04T09:06:56.830 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:06:56.831 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:06:56.831 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2023-12-04T09:06:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:06:56.834 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-04T09:06:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:06:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:06:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2023-12-04T09:06:56.839 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:06:56.839 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-04T09:06:56.839 INFO:teuthology.orchestra.run.smithi006.stderr:09:06:56 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-04T09:07:56.893 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:56.893 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-04T09:07:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-04T09:07:56.994 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:56.994 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-12-04T09:07:56.995 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:56.996 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-12-04T09:07:56.997 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:56.998 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:56.998 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2023-12-04T09:07:57.017 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.017 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/$(basename ${i}).smithi006.front.sepia.ceph.com ; done 2023-12-04T09:07:57.022 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.022 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.023 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-12-04T09:07:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - WARNING - cbt - sudo killall -9 rados 2023-12-04T09:07:57.049 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - WARNING - cbt - error 1 seen, continuing anyway... 2023-12-04T09:07:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d. 2023-12-04T09:07:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-12-04T09:07:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.070 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-12-04T09:07:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.089 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2023-12-04T09:07:57.092 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.092 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-04T09:07:57.093 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2023-12-04T09:07:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-04T09:07:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-04T09:07:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-12-04T09:07:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:57.404 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:57 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-12-04T09:07:58.044 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:58 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:07:58.044 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:58 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:07:58.044 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-12-04T09:07:59.049 INFO:teuthology.orchestra.run.smithi006.stderr:09:07:59 - INFO - cbt - Checking Health after pool creation. 2023-12-04T09:08:04.054 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:04 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:04.054 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:04 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:04.054 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-04T09:08:04.374 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:04 - INFO - cbt - Final Pool Health Check. 2023-12-04T09:08:09.380 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.380 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.380 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-04T09:08:09.696 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.696 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-04T09:08:09.726 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.726 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-04T09:08:09.773 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.773 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-12-04T09:08:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.774 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-12-04T09:08:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.776 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-12-04T09:08:09.873 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:09.874 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:09 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-12-04T09:08:10.557 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:10.557 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:10.557 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-12-04T09:08:10.832 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:10.832 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:10.833 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2023-12-04T09:08:10.918 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:10.918 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:10.918 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:10 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2023-12-04T09:08:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.687 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.687 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2023-12-04T09:08:11.689 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.689 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-04T09:08:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2023-12-04T09:08:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-04T09:08:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-04T09:08:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - INFO - cbt - Running radosbench write test. 2023-12-04T09:08:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2023-12-04T09:08:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:08:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:09:08:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2023-12-04T09:09:12.826 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:12.826 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-04T09:09:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:12.880 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-04T09:09:12.936 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:12.936 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2023-12-04T09:09:12.938 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:12.938 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2023-12-04T09:09:12.939 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:12.939 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:12.939 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2023-12-04T09:09:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.273 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.273 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2023-12-04T09:09:13.292 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.292 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/$(basename ${i}).smithi006.front.sepia.ceph.com ; done 2023-12-04T09:09:13.321 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - INFO - cbt - Convert results to json format. 2023-12-04T09:09:13.321 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.322 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.322 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.322 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-12-04T09:09:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-12-04T09:09:13.866 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.866 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:13.866 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-12-04T09:09:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq 2023-12-04T09:09:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.168 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/ceph_settings.out 2023-12-04T09:09:14.903 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.904 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.904 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/collectl 2023-12-04T09:09:14.906 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.906 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-04T09:09:14.908 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.908 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.908 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - INFO - cbt - Running radosbench seq test. 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.911 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/output.0 2023-12-04T09:09:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/output.1 2023-12-04T09:09:29.041 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.041 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-04T09:09:29.089 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.090 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-04T09:09:29.141 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.141 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf/perf.data 2023-12-04T09:09:29.142 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.143 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.* 2023-12-04T09:09:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.144 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/historic_ops.out 2023-12-04T09:09:29.453 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.464 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.464 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/* 2023-12-04T09:09:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - get_fqdn_local()=smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/$(basename ${i}).smithi006.front.sepia.ceph.com ; done 2023-12-04T09:09:29.511 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - INFO - cbt - Convert results to json format. 2023-12-04T09:09:29.511 INFO:teuthology.orchestra.run.smithi006.stderr:09:09:29 - DEBUG - cbt - Nodes : ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.547 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-12-04T09:09:29.557 DEBUG:teuthology.orchestra.run.smithi006:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-12-04T09:09:29.699 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-12-04T09:09:29.699 INFO:tasks.cbt_performance:reading cbt preferences 2023-12-04T09:09:29.699 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-12-04T09:09:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.753 INFO:teuthology.orchestra.run.smithi006.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.smithi006.front.sepia.ceph.com 2023-12-04T09:09:29.754 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:09:29.754 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.smithi006.front.sepia.ceph.com of=/dev/stdout 2023-12-04T09:09:29.786 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-04T09:09:29.787 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2023-12-04T09:09:29.841 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-04T09:09:29.841 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-04T09:09:29.842 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:09:29.842 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.smithi006.front.sepia.ceph.com of=/dev/stdout 2023-12-04T09:09:29.897 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-04T09:09:29.897 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2023-12-04T09:09:29.952 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-04T09:09:29.952 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-04T09:09:29.952 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:09:29.953 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.smithi006.front.sepia.ceph.com of=/dev/stdout 2023-12-04T09:09:30.007 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-04T09:09:30.008 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2023-12-04T09:09:30.063 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-04T09:09:30.063 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-04T09:09:30.063 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-12-04T09:09:30.063 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.smithi006.front.sepia.ceph.com of=/dev/stdout 2023-12-04T09:09:30.119 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-04T09:09:30.119 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2023-12-04T09:09:30.174 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-04T09:09:30.174 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-04T09:09:30.182 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-04T09:09:30.189 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-04T09:09:30.195 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-04T09:09:30.201 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-04T09:09:30.201 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-04T09:09:30.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-04T09:09:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:09:30.318+0000 7f92f179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:09:30.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-12-04T09:09:30.325+0000 7f92f179e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-04T09:09:30.528 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-04T09:09:30.528 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-04T09:09:30.528 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-04T09:09:30.528 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T09:09:34.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-04T09:09:36.629 INFO:tasks.ceph.osd.0:Stopped 2023-12-04T09:09:36.630 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-04T09:09:36.630 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T09:09:39.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-04T09:09:39.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-04T09:09:42.732 INFO:tasks.ceph.osd.1:Stopped 2023-12-04T09:09:42.732 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-04T09:09:42.732 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T09:09:45.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-04T09:09:45.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-04T09:09:48.833 INFO:tasks.ceph.osd.2:Stopped 2023-12-04T09:09:48.834 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-04T09:09:48.834 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-04T09:09:48.834 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T09:09:48.866 INFO:tasks.ceph.mgr.x:Stopped 2023-12-04T09:09:48.866 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-04T09:09:48.867 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-04T09:09:48.867 INFO:teuthology.orchestra.run:waiting for 300 2023-12-04T09:09:48.920 INFO:tasks.ceph.mon.a:Stopped 2023-12-04T09:09:48.920 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-04T09:09:48.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-12-04T09:09:49.026 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:49.026 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-04T09:09:49.240 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:49.241 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-04T09:09:49.320 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:09:49.320 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-04T09:09:49.403 INFO:tasks.ceph:Archiving mon data... 2023-12-04T09:09:49.404 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785/data/mon.a.tgz 2023-12-04T09:09:49.404 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-12-04T09:09:49.420 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.fCy90rYomD 2023-12-04T09:09:49.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.fCy90rYomD 2023-12-04T09:09:49.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.fCy90rYomD 2023-12-04T09:09:49.654 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.fCy90rYomD is 179KB 2023-12-04T09:09:49.678 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.fCy90rYomD 2023-12-04T09:09:49.692 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-04T09:09:49.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-04T09:09:49.832 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-04T09:09:49.833 INFO:tasks.ceph:Archiving crash dumps... 2023-12-04T09:09:49.834 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785/remote/smithi006/crash 2023-12-04T09:09:49.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-04T09:09:49.902 INFO:tasks.ceph:Compressing logs... 2023-12-04T09:09:49.902 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-04T09:09:49.980 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83060.log 2023-12-04T09:09:49.980 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84538.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82101.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83060.log: /var/log/ceph/ceph-client.admin.84538.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82834.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82061.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82979.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83061.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83273.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83525.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83357.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84538.log.gz 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82101.log.gz 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82141.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84463.log 2023-12-04T09:09:49.985 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83483.log 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83059.log 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83060.log.gz 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.83483.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82061.log.gz 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83273.log.gz 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84934.log 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83059.log: 50.0%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.82141.log.gz 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84934.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80492.log 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.84934.log.gz 2023-12-04T09:09:49.986 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82792.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph.tmp-client.admin.80492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82329.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80492.log.gz 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84401.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82792.log: /var/log/ceph/ceph-client.admin.82329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83293.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83187.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.83293.log: -- /var/log/ceph/ceph-client.admin.81863.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82937.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.81863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83186.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83483.log.gz 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84602.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82736.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83293.log.gz 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82299.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82736.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-04T09:09:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.82299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82895.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84463.log.gz 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83187.log.gz 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81863.log.gz 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81951.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mgr.x.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83186.log.gz 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.82895.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83147.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr: 65.3%/var/log/ceph/ceph-client.admin.81951.log: -- replaced with /var/log/ceph/ceph-client.admin.84602.log.gz 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84515.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81951.log.gz 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83147.log.gz/var/log/ceph/ceph-osd.0.log: 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84515.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85235.log 2023-12-04T09:09:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83377.log 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84603.log 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.85235.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.85235.log.gz 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84492.log 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.84603.log: /var/log/ceph/ceph-client.admin.84935.log 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83441.log 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84492.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84515.log.gz 2023-12-04T09:09:49.994 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.84935.log: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.84935.log.gz 2023-12-04T09:09:49.997 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.83441.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82736.log.gz 2023-12-04T09:09:49.997 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82792.log.gz 2023-12-04T09:09:49.997 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82834.log.gz 2023-12-04T09:09:49.997 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-04T09:09:49.997 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82895.log.gz 2023-12-04T09:09:49.997 INFO:teuthology.orchestra.run.smithi006.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82937.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82979.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83061.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83059.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83357.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84401.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83377.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.84603.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83441.log.gz 2023-12-04T09:09:49.998 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83525.log.gz 2023-12-04T09:09:50.018 INFO:teuthology.orchestra.run.smithi006.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82299.log.gz 2023-12-04T09:09:50.019 INFO:teuthology.orchestra.run.smithi006.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82329.log.gz 2023-12-04T09:09:50.036 INFO:teuthology.orchestra.run.smithi006.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84492.log.gz 2023-12-04T09:09:50.036 INFO:teuthology.orchestra.run.smithi006.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-04T09:09:50.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-04T09:09:50.122 INFO:teuthology.orchestra.run.smithi006.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-04T09:09:58.554 INFO:teuthology.orchestra.run.smithi006.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-04T09:09:58.791 INFO:teuthology.orchestra.run.smithi006.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-04T09:09:58.818 INFO:teuthology.orchestra.run.smithi006.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-04T09:09:58.819 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-12-04T09:09:58.819 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m8.862s 2023-12-04T09:09:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m5.150s 2023-12-04T09:09:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.272s 2023-12-04T09:09:58.820 INFO:tasks.ceph:Archiving logs... 2023-12-04T09:09:58.820 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785/remote/smithi006/log 2023-12-04T09:09:58.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-04T09:09:59.557 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-04T09:09:59.566 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-04T09:09:59.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-04T09:09:59.636 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-04T09:09:59.644 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-04T09:09:59.644 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T09:09:59.658 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2023-12-04T09:09:59.719 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-04T09:09:59.720 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2023-12-04T09:09:59.720 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +625us[ +625us] +/- 84ms 2023-12-04T09:09:59.720 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -687us[ -687us] +/- 74ms 2023-12-04T09:09:59.720 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +184us[ +185us] +/- 85ms 2023-12-04T09:09:59.720 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-04T09:09:59.732 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-04T09:09:59.740 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-04T09:09:59.740 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-04T09:09:59.749 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-04T09:09:59.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-04T09:09:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T09:09:59.965 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-04T09:09:59.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-04T09:09:59.982 INFO:teuthology.task.internal:Duration was 1224.762295 seconds 2023-12-04T09:09:59.982 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-04T09:10:00.011 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-04T09:10:00.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-04T09:10:00.186 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-04T09:10:00.496 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-04T09:10:00.496 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2023-12-04T09:10:00.497 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-04T09:10:00.517 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-04T09:10:00.517 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-04T09:10:00.589 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-04T09:10:00.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-04T09:10:00.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-04T09:10:00.753 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-04T09:10:00.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-04T09:10:00.795 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-04T09:10:00.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-04T09:10:00.897 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2023-12-04T09:10:00.940 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-04T09:10:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T09:10:00.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-04T09:10:00.963 INFO:teuthology.task.internal:Transferring archived files... 2023-12-04T09:10:00.963 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-12-04_08:39:13-crimson-rados-wip-matanb-crimson-submit-error-cleanup-testing-distro-crimson-smithi/7476785/remote/smithi006 2023-12-04T09:10:00.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-04T09:10:01.080 INFO:teuthology.task.internal:Removing archive directory... 2023-12-04T09:10:01.080 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-04T09:10:01.125 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-04T09:10:01.138 INFO:teuthology.task.internal:Not uploading archives. 2023-12-04T09:10:01.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-04T09:10:01.147 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-04T09:10:01.147 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-04T09:10:01.181 INFO:teuthology.orchestra.run.smithi006.stdout: 265434 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 4 09:10 /home/ubuntu/cephtest 2023-12-04T09:10:01.182 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-04T09:10:01.192 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-04T09:10:01.213 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-submit-error-cleanup-testing&var-job_id_selected=7476785 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} duration: 1224.762294769287 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-12-04T09:10:01.214 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T09:10:01.286 INFO:teuthology.run:pass